Active Audition and Sensorimotor Integration for Sound Source - PowerPoint PPT Presentation

Active Audition and Sensorimotor Integration for Sound Source Localization Mathieu Bernard 25 novembre 2011 1/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Introduction CIFRE thesis. Co-direction : Patrick Pirim, Brain Vision Systems Bruno Gas, ISIR, UPMC Alain de Cheveign´ e, LPP, Paris Descartes Outline Artificial auditory system for sound localization, Audio-tactile model for texture recognition, Active audition and sensorimotor integration. 2/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Autonomous robotics Psikharpax, the robot rat 3/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Artificial auditory system Bioinspired sound localization Outer ear Inner ear Binaural localiation cues ITD - ILD Outer ear Inner ear Auditory model Implementation robotic robot simulation sound simulated capture sound sources core library real time - c++ wav files standalone programs 4/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Outer ear model Pinna Auditory fovea Microphone Oriented toward the fovea Support Servomotor Outer ear = Pinna, microphone and software capture. Spectral and directional cues. 5/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Inner ear model (1/2) Inner ear = cochlear model and pulse train generation. Cochlear model Gammatone filterbank Usually 30 channels from 300Hz to 8kHz at 20kHz 6/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Inner ear model (2/2) Inner ear = cochlear model and pulse train generation. Pulse train generation Cochlear channel output ⇒ Pulse train Discrete representation, noise suppression. A pulse = temporal and amplitude information. 7/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

ILD computation (1/2) Energy for each pulse p ( t ), the energy is : u = t � p ( u ) 2 E ( t ) = (1) u = t − T 8/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

ILD computation (2/2) ILD = Interaural Level Difference 2 E left , i ( t ) For each channel i : ILD i ( t ) = E left , i ( t )+ E right , i ( t ) − 1 9/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

ITD computation (1/3) ITD = Onset extraction and delay lines. Onset extraction from a pulse train Comparison with a dynamic threashold, 2 parameters. 10/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

ITD computation (2/3) ITD = Onset extraction and delay lines. Delay lines model 11/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

ITD computation (3/3) ITD = Onset extraction and delay lines. 12/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Auditory and tactile model for texture perception (1/3) Similar model for transduction and processing Audition and touch support fine texture discrimination skills. Rat vibrissae and cochlea transduction based on resonance. Strong interaction for auditory/tactile spectral processing. 13/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Auditory and tactile model for texture perception (2/3) Whiskers filterbank adapted from gammatone cochlear model : Feature extraction = Instantaneous Mean Power : 14/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Auditory and tactile model for texture perception (3/3) Texture classification : 8 textures, 3 layer perceptron Response to a pure tone (whisker at 630 Hz) Influence of the number of channels 15/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Auditory Evoked Behaviors (1/2) Outer ear Inner ear Energy ILD Outer ear Inner ear Energy Auditory model Motor control Neck and body Motor control Neck ⇒ ILD minimization Wheels ⇒ Follow neck orientation ILD < 0 ⇒ turn right, Constant speed, ILD > 0 ⇒ turn left. Smooth rotations. Video ! Video ! 16/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Auditory Evoked Behaviors (2/2) Phonotaxis trajectories. Front-back disambiguation. 17/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Sensorimotor Approach (1/3) Environment state e ∈ E and motor state m ∈ M , Sensory state s ∈ S , we have s = Φ( m , e ), Φ is called a sensorimotor law. 18/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Sensorimotor Approach (2/3) Quand on dit [...] que nous localisons tel objet en tel point de l’espace [...] cela signifie simplement que nous nous repr´ esentons les mouvements qu’il faut faire pour atteindre cet objet. [...] Nous nous repr´ esentons les sensations musculaires qui accompagnent [ces mouvements] et qui n’ont aucun caract` ere g´ eom´ etrique, qui par cons´ equent n’impliquent nullement la pr´ eexistance de la notion d’espace. H. Poincar´ e, L’espace et la g´ eom´ etrie, 1845. Proposed formalization for localization Find the motor state ˜ m such as m = argmin ˜ | Φ( m , e ) − Φ( m 0 , e 0 ) | . m ∈M 19/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Sensorimotor Approach (3/3) Sound source localization Two ways for ˜ m estimation Orienting behavior After completion we have ˜ m = m end s end = Φ( m end , e ) = Φ( m 0 + δ m , e 0 + δ e ) , Manifold learning Sensory space S lies on a low-dim manifold R , Dimension reduction technique : S → R , Same topology as the embodying space (at least locally). 20/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Manifold learning (1/3) CAMIL database, ILD vectors : 21/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Manifold learning (2/3) CAMIL database, ITD vectors : d=690 → d=3 22/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Manifold learning (3/3) 2 outer ears models : HRTF and directive filters Front-back disambiguation with spectral cues : 23/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Sensorimotor integration (1/2) Learning algorithm Iterative learning of R , Self-supervised. 24/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Sensorimotor integration (2/2) Simulation results over 400 experiments 25/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Intrinsic Dimension Estimation Simulation (ILD, azimtuh 180 & 360) CAMIL dataset (ILD & ITD, azimtuh 360, elevation [-30, 30]) 26/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Perspectives SensoriMOTOR 27/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration