Neural network representations and visual processing in brains - PowerPoint PPT Presentation

Neural network representations and visual processing in brains Katja Seeliger [ www.ccnlab.net | Marcel van Gerven ]

Neural network representations and visual processing in brains Computational Cognitive Neuroscience Lab Develop computational models of cognitive processes (sensory, attention, control, language …) Connectionist modeling with artificial neural networks Validation with neural / behavioural data www.ccnlab.net

Neural network representations and visual processing in brains Representation of visual processing ? Neural Natural world representation www.ccnlab.net

Neural network representations and visual processing in brains Biological visual system basics: Striate cortex "The receptive field of an individual sensory neuron is the particular region of the sensory space in which a stimulus will trigger the firing of that neuron." (Sherrington, 1906) Specific firing for: Location, stimulus orientation, colour, right / left eye (…?) Sherrington (1906) Hubel, D. H., & Wiesel, T. N. (1968).

Neural network representations and visual processing in brains Basics: Learning of feature detectors Feature detectors in the early visual cortex don’t exist a-priori , but are learned from exposure to the visual environment. —> There must be a learning algorithm. Colin Blakemore and Grahame F Cooper (1970). “Development of the brain depends on the visual environment”. In: Nature 228, pp. 477 –478.

Neural network representations and visual processing in brains Basics: Hierarchy, local regions Visual system regions (simplified)

Neural network representations and visual processing in brains Basics: Two streams-hypothesis Dorsal stream "Where? How?" Localization and motion perception Ventral stream "What?" Object recognition Image source: https://en.wikipedia.org/wiki/Two-streams_hypothesis

Neural network representations and visual processing in brains Hand-crafted (and shallow) representations Gabor edge detectors ? ≈ Fourier powers Encoding in visual alphabet Object categories ( feature space )

Neural network representations and visual processing in brains Great shallow representations can get you quite far… Feature space: Gabor filter pyramid + motion energy Shinji Nishimoto et al. (2011). “Reconstructing visual experiences from 
 brain activity evoked by natural movies”. In: Current Biology 21.19.

Neural network representations and visual processing in brains Method: Voxel-wise encoding models natural stimuli brain activity Naselaris, Thomas, et al. "Encoding and decoding in fMRI." Neuroimage 56.2 (2011): 400-410.

Neural network representations and visual processing in brains Method: Voxel-wise encoding models encode predict Naselaris, Thomas, et al. "Encoding and decoding in fMRI." Neuroimage 56.2 (2011): 400-410.

Neural network representations and visual processing in brains Method: Voxel-wise encoding models Based on the chosen Model performance encoding / feature space : Correlation[ predicted , … we predict the activity for measured ] 
 each voxel using regularized on the test set per voxel (map) linear regression . Naselaris, Thomas, et al. "Encoding and decoding in fMRI." Neuroimage 56.2 (2011): 400-410.

Neural network representations and visual processing in brains Deep representations Deep convolutional Visual system regions neural network (simplified)

Neural network representations and visual processing in brains Deep representations: AlexNet first layer Specific for: Location, stimulus orientation, colour, colour opponency.. Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012.

Neural network representations and visual processing in brains Deep representations: AlexNet convolutional hierarchy Güçlü, U., & van Gerven, M. A. (2015). Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. The Journal of Neuroscience, 35(27), 10005-10014.

Neural network representations and visual processing in brains Deep representations across ventral stream Relating fMRI responses to static images and VGG-S layers (via predictive voxel-wise linear models (encoding models) ) Güçlü, U., & van Gerven, M. A. (2015). Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. The Journal of Neuroscience, 35(27), 10005-10014.

Neural network representations and visual processing in brains Low and high deep convolutional layers in time Relating MEG responses to static images and VGG-S layers in time (via predictive voxel-wise linear models (encoding models) ) K. Seeliger, M. Fritsche, U. Güçlü, S. Schoenmakers, J.-M. Schoffelen, S. E. Bosch, M. A. J. van Gerven (2017: CNN- based Encoding and Decoding of Visual Object Recognition in Space and Time (bioRxiv , and in press )

Neural network representations and visual processing in brains Assignment of VGG-S layers to anatomy K. Seeliger, M. Fritsche, U. Güçlü, S. Schoenmakers, J.-M. Schoffelen, S. E. Bosch, M. A. J. van Gerven (2017: CNN- based Encoding and Decoding of Visual Object Recognition in Space and Time (bioRxiv , and in press )


 Neural network representations and visual processing in brains Deep representations across dorsal stream Relating fMRI responses to movies and action recognition DNN 
 > spatiotemporal receptive fields U Güçlü, MAJ van Gerven (2015): Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. NeuroImage

Neural network representations and visual processing in brains Visual cortex == 8 representations from AlexNet? Macaque visual system connections and hierarchy by van Essen (1991) Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex, 1(1), 1-47.

Presentation Title Neural network representations and visual processing in brains Questions?