Modelling of sensory integration with neural network systems Lennart Gustafsson, Andrew Paplinski & Tamas Jantvik
Q: Why integrate sensory information? A: Because biology does it, at least for higher order animals, and the animals gain from it… “… the major functions of multisensory convergence and integration seem aimed at enhancing the detection of behaviourally-relevant stimuli, and of promoting rapidity of behavioural responding (e.g. motoric orienting). We would add cognitive processing (e.g. attentional orienting and cognition) to the list of functions that benefit from multisensory processing.” From Schroeder et al.: “Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing”, Int. J. of Psychophysiology, 2003 Thus, in biology, sensory integration of congruent stimuli can yield: • Shorter reaction time to an event • Lower threshold for detecting an event (e.g. integrated sensory information give larger responses and/or make additional neurons active) • Better (faster) learning of how to handle an event. • The resulting multimodal percept is much more robust against corrupted stimuli (against ‘noise’) than the individual unimodal percepts. • Modulation of unimodal percepts using the multimodal representation. All these advantages apply to situations where sensory stimuli in different modalities are congruent, i.e. they are different aspects of the same event.
Example: Response time to stimuli (in cortex) From Laurienti et al.: “Semantic congruence is a critical factor in multisensory behavioural performance”, Experimental Brain Research, 2004 Response for congruent visual and auditory stimuli is quicker than for the corresponding unimodal stimulation.
Exmaple: Multisensory integration can be a very early event Multisensory interaction in a single neuron in the superior colliculus (located in the midbrain of mammals). Information flows through here at a very early stage of sensory processing, before cortical processing. Sensory integration can thus increase the response to an event. The picture shows relative response levels to visual stimuli, auditory and the combination of the two. From King & Calvert: “Multisensory integration: Perceptual grouping by eye and ear”, Current Biology, 2001
A: … and it can develop in an automatic way Example: Multisensory integration in neocortex From Beauchamp: “See me, hear me, touch me: multisensory integration in lateral occipital- temporal cortex”, Current Opinion in Neurobiology, 2005
A closer look at responses of neurons From Beauchamp et al.: “Unraveling multisensory integration: patchy organization within human STS multisensory cortex”, Nature Neuroscience, 2004 The neocortex develops automatically by learning from incoming stimuli, so it wouldn’t be unreasonable to assume that this integration mechanism has also been developed automatically.
Bimodal integration of auditory and visual stimuli What if we could mimic the brain’s way of sensory integration and use it in engineering applications. We’ll start with modelling this integration process of visual letters and speech sounds. From van Atteveldt et al. ”Integration of Letters and Speech Sounds in the Human Brain”, Neuron, 2004
An attempt of mimicry We want to create an architecture that features as many of the advantageous properties of those found in biological sensory integration of letters and phonemes as possible. It should: • Communicate with the outside world via input and output signals • Support integration of congruent signals • Increase the robustness of the integrated signal against corruption/noise • Modulate unimodal sensory signals with the integrated one to converge to a coherent state • Be tuned using the input signals (i.e. support somewhat automatic setup) Aiming to use the architecture in engineering application it should also be: • Fast, at least during application (application as opposed to training) • Versatile And, in an initial approach it is always wise to keep things simple.
Step 1: A way of integration A bimodal, hierarchical system, consisting of three Kohonen maps connected in a bottom-up, or feed-forward, fashion. The networks output the coordinate y bm of the winner neuron. Bimodal map is trained with 2 concatenated winner coordinates of congruent stimuli ( x lt , x ph ). v V W We call this a MuSON; Multimodal SOM bm Self-Organizing Network. x Letters: 23-element vector y y lt ph 2 2 based on principal component analysis of > 1000 pixel v v patterns representing letters V W V W SOM lt SOM ph Phonemes: 36-element vector x x consisting of Mel frequency cepstral coefficients of n lt n ph average phonemes as spoken x x by ten Swedish speakers lt ph
Step 1 results Bimodal Kohonen map when ‘o’ is the input to the system. Notice that there is high activation of neighbouring populations as well. This is unbiological – lateral inhibition is lacking in the Kohonen map. A patch thus consists of a group of neurons that give the highest activity for a stimulus. The winner neuron for a stimulus is the one with the maximal response to that stimulus.
Step 1 results Patches of maximum activity in response to stimuli after self-organization (An example, organization may change from one run to another, but the relations between patches adhering to similar stimuli are always there). Noteworthy things: • Letter map: Positions of ‘i’ and ‘l’, ‘a’, ‘ä’ and ‘å’, and other look-alikes. • Phoneme map: Position of ‘p’ and ‘t’, ‘n’ and ‘m’, and other sound-alikes. • Bimodal map: Has a topology too; of coordinates. For instance: stimuli adjacent in both sensory maps are adjacent in the bimodal map, ‘f’ and ‘v’, ‘i’ and ‘l’. Phoneme map Bimodal map (4D) Letter map y v g a ä a r ö ä s y ä S e u å f å ö d f s n e e i l u S v o l r m y t i m ö b g n d i o p b t d S t l u å v k b p n m k g f s k p o r a
Step 1 results Activity in Kohonen maps for three different inputs. Solid lines: Perfect letters Dash-dot lines: Heavily corrupted letters
Step 1 results Activity in Kohonen maps for three different inputs. Solid lines: Perfect letters Dash-dot lines: Heavily corrupted letters Conclusion: Some robustness has been achieved.
Step 2: Modelling feedback using a MuSON with feedback y bm 2 Continuing to be influence by the findings W V v,d Feedback and interpretations of van Atteveldt et al in SOM bm ”Integration of Letters and Speech Sounds x in the Human Brain” we add feedback from y the bimodal SOM to a SOM in the speech rph sound processing system that merges 2 together the sensory input with the feedback (again, by concatenations of W V v,d coordinates). SOM rph x The sensory and the bimodal maps are trained first, as before, and then the map 2 y y 2 lt ph 2 dealing with feedback. W v,d V W v,d V In application SOM rph is passed through during the first loop. SOM lt SOM ph x x n lt n ph x x lt ph
Step 2 results A complete set of maps after self-organization of MuSON with feedback Letter map Phoneme map Re−coded phoneme map a g m ä S s ä S å n e y s e y ä i f ö ö e i f u d l S u l u b s b d d y ö f v n m v n m r v r g t o g t r o t o a å p k i l k b p a å p k Bimodal map f r t k i The organization of the two phoneme maps are very similar s l p – it is as it should be. Otherwise it wouldn’t work very well, y b would it? S o ö v e ä d u a m å n g
Step 2 results The processing of three corrupted phonemes in MuSON with feedback Letter map Phoneme map Re−coded phoneme map m å i i m m i å å Conclusion: Modulation of unimodal sensory signal with the integrated one to reach to a coherent state is modelled. “Assuming the activity in auditory cortex generally corresponds to a perceptual experience of something heard, a likely function of a converging visual or somatosensory input would be to enhance auditory analysis of that stimulus. … but the perceptual experience would remain auditory .” From Schroeder et al.: “Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing”, Int. J. of Psychophysiology, 2003.
Step 3: Making the activity levels matter Phoneme map Relaxation loop 0 A response from a SOM to an input that is 1 unrecognized. There is some activity and whence 0.8 a maximum activity, and a winner neuron. But 0.6 these activities are low. 0.4 0.2 If the activity levels are allowed to matter in the 0 architecture the model becomes fuller. 10 10 20 20 30 30
Step 3: Making the activity levels matter a bm v bm 2 1 v, a V SumSOM Type2 Goal: If the auditory processing (for example) yields low activity its output Bi−modal should have low significance in the bi- Processing v, a v, a Feedback modal processing. Idea: When combining coordinates of a v ph ph winner neurons, make the combination 2 1 weighted using the activities of these a v v, a lt lt neurons. V 2 1 SumSOM Type1 v, a v – position of the winner. V W Auditory a – activity level of the winner. SOM lt Processing v, a x x n n ph 1 lt 2 x x lt ph
Recommend
More recommend
