Cognitive computational neuroscience of vision Nikolaus - PowerPoint PPT Presentation

Cognitive science Computational neuroscience Cognitive computational neuroscience of vision Nikolaus Kriegeskorte Department of Psychology, Department of Neuroscience Zuckerman Mind Brain Behavior Institute Affiliated member, Electrical Engineering, Columbia University Artificial intelligence

Cognitive computational neuroscience Cognitive science Computational neuroscience neural A common language for network expressing theories about models brain information processing Artificial intelligence Kriegeskorte & Douglas 2018

How can we test neural network models with brain-activity data?

Predicting representational spaces responses responses activity profile activity profile stimuli stimuli activity pattern activity pattern ? activity patterns ... ... model brain experimental ... stimuli Diedrichsen & Kriegeskorte 2017, Kriegeskorte & Diedrichsen 2019

Predicting representational spaces responses activity profile stimuli activity pattern Diedrichsen & Kriegeskorte 2017, Kriegeskorte & Diedrichsen 2019

Predicting representational spaces model distance matrix features weights encoding representational similarity model analysis responses L2 weight penalty activity profile stimulus 1 stimuli stimulus 3 response 3 activity pattern activity activity response 1 (e.g. neuron, voxel) Diedrichsen & Kriegeskorte 2017, Kriegeskorte & Diedrichsen 2019

Predicting representational spaces model each response model stimulus-by-stimulus matrix separately of summary statistics model second-moment matrix distance matrix features weights encoding pattern component representational similarity model model analysis L2 weight penalty stimulus 1 stimulus 3 stimulus 3 response 3 activity activity activity response 1 (e.g. neuron, voxel) Core commonality : All three test hypotheses about the model representational model activity-profiles distribution second moment of the activity profiles. distances Diedrichsen & Kriegeskorte 2017, Kriegeskorte & Diedrichsen 2019

The onion of brain representations information potentially used information potentially extracted by researchers by single readout neurons 1 spatially organized neuronal population code (neuronal locations and activity profiles L U ) 4 total encoded information (downstream neuron can perform arbitrary linear or nonlinear readout from all neurons) 2 activity-profiles distribution U (activity profiles or all moments 5 linear neuronal readout of activity-profiles distribution) (downstream neuron can perform linear readout from all neurons) 3 representational geometry 6 restricted-input linear readout nd G (2 moment of activity profiles or (downstream neuron can perform linear D representational distance matrix ) readout from a limited number of neurons) 7 local linear readout (downstream neuron can perform linear or radial-basis readout from neurons in a restricted spatial neighborhood) researcher information encoded information comprehensive focused implicit explicit Kriegeskorte & Diedrichsen 2019

The onion of brain representations information potentially used information potentially extracted by researchers by single readout neurons 1 spatially organized neuronal population code (neuronal locations and activity profiles L U ) 4 total encoded information (downstream neuron can perform arbitrary linear or nonlinear readout from all neurons) 2 activity-profiles distribution U (activity profiles or all moments 5 linear neuronal readout of activity-profiles distribution) (downstream neuron can perform linear readout from all neurons) 3 representational geometry 6 restricted-input linear readout nd G (2 moment of activity profiles or (downstream neuron can perform linear D representational distance matrix ) readout from a limited number of neurons) 7 local linear readout (downstream neuron can perform linear or radial-basis readout from neurons in a restricted spatial neighborhood) researcher information encoded information comprehensive focused implicit explicit Kriegeskorte & Diedrichsen 2019

Representational similarity analysis stimuli stimuli 000 0 000000 0 0 00000 0 000000 stimuli stimuli representational ! dissimilarity matrix (RDM) dissimilarity (e.g. crossvalidated Mahalanobis distance estimator) ? activity patterns ... ... model brain experimental ... stimuli Kriegeskorte et al. 2008

Representational feature weighting with non-negative least-squares model RDM f 1 f 2 . . . f k weighted-model w 1 f 1 w 2 f 2 . . . w k f k RDM

Representational feature weighting with non-negative least-squares predicted distance model feature k weight k 𝑜 2 = ෍ መ 𝑙 𝑘 ] 2 𝑒 𝑗,𝑘 [𝑥 𝑙 𝑔 𝑙 𝑗 − 𝑥 𝑙 𝑔 2  feature-1 RDM w 1 2  feature-2 RDM + w 2 𝑙=1 stimuli i , j 𝑜 ... 2 ∙ [𝑔 𝑙 𝑘 ] 2 = ෍ 𝑥 𝑙 𝑙 𝑗 − 𝑔 𝑙=1 2  feature-k RDM The squared distance RDM + w k of weighted model features = weighted-model RDM equals a weighted sum of single-feature RDMs. 2 𝑜 𝑒 2 − ෍ 𝑥 𝑙2 ∙ RDM 𝑙 2 2 − መ = arg min 𝐱∈𝐒 +𝒐 ෍ 2 𝐱 = arg min 𝐱∈𝐒 +𝒐 ෍ 𝑒 𝑗,𝑘 𝑒 𝑗,𝑘 𝑗≠𝑘 𝑙=1 𝑗,𝑘 𝑗≠𝑘 w k weight given to model feature k f k (i) model feature k for stimulus i d i,j distance between stimuli i,j w is the weight vector [ w 1 w 2 ... w k ] that minimizes the sum of squared errors

Deep convolutional networks predict IT representational geometry model comparisons (stimulus bootstrap, p < 0.05, Bonferroni corrected for all pairwise comparisons) 0.7 highest accuracy any model can achieve 0.6 noise ceiling 0.5 accuracy of human IT 0.4 other subjects’ average dissimilarity matrix as model prediction 0.3 [group-average of Spearman’s  ] accuracy below noise ceiling 0.2 performance range of p < 0.05, Bonf. corr. (stimulus bootstrap) computer-vision features 0.1 accuracy above 0 SE p < 0.05, Bonf. corr. (stimulus bootstrap) 0 (stimulus bootstrap) weighted combination of layers and SVM discriminants fully convolutional connected Khaligh-Razavi & Kriegeskorte 2014, Nili et al. 2014 (RSA Toolbox), Storrs et al. (in prep.)

Do recurrent neural networks provide better models of vision? Courtney Spoerer

Recurrent networks can recycle their limited computational resources over time. This might boost the performance of a physically finite model or brain. Kriegeskorte & Golan 2019

Layer 1 lateral connectivity is consistent with primate V1 connectivity RCNN, layer 1, lateral connectivity templates (first 5 principal components) Spoerer et al. pp2019

Recurrent models can trade off speed of computation for accuracy recurrent convolutional accuracy feedforward models computational cost [floating-point operations × 10 11 ] Spoerer et al. pp2019

Recurrent models can trade off speed of computation for accuracy recurrent convolutional accuracy feedforward models computational cost [floating-point operations × 10 11 ] Spoerer et al. pp2019

RCNN reaction times tend to be slower for images humans are uncertain about correlation between human certainty and RCNN reaction time [Spearman]

Tim Kietzmann Can recurrent neural network models capture the representational dynamics in the human ventral stream?

Fitting model representational dynamics with deep representational distance learning McClure & Kriegeskorte 2016 Task: find an image-computable network to model the first 300ms of representational dynamics of the ventral stream.

Recurrent models better explain representations and their dynamics magnetoencephalography functional magnetic resonance imaging feedforward recurrent Recurrent networks significantly outperform ramping feedforward models in predicting ventral-stream representations (MEG and fMRI).

How can we build neural network models of mind and brain? big world big brain big behavioral data data data Divergent: Exploring the space of computational big models models with world data Convergent: Constraining models • Training with brain and behavioral data • different sets of stimuli • • inferential model selection (model different tasks • Units parameters learned for a task) • stochasticity • reweighting of units • context-modulation • linear remixing of units • Architecture • deep learning of model parameters • skipping connections from brain-activity data • recurrent connections