From brain responses to algorithms: advances in parsing the computational architecture of perceptual decision making with MEG and machine learning Laura Gwilliams & Jean-Rémi King 12th October 2018 Laura Gwilliams | New York University | @GwilliamsL
The world is an uncertain place ❖ Noise ❖ Ambiguity Laura Gwilliams | New York University | @GwilliamsL
AI can categorise, too ❖ Artificial intelligence has sought to solve a similar problem in visual processing ❖ Deep neural networks (DNNs) can label images very accurately Laura Gwilliams | New York University | @GwilliamsL
AI and neural convergence ❖ Correspondence has been found in terms of the representations employed by brains and DNNs Yamins et al., 2014 Laura Gwilliams | New York University | @GwilliamsL
AI and neural convergence ❖ Not so surprising, given that aspects of DNNs are modelled on vision neuroscience ❖ There is more to characterising a system than simply knowing the representations it uses: ❖ Architecture ❖ Computation Laura Gwilliams | New York University | @GwilliamsL
Research Question What is the computational architecture of perceptual decision making? Laura Gwilliams | New York University | @GwilliamsL
Roadmap What is the order of operations performed on the sensory input? What are the underlying computations at the decision stage? How are the stages linked to one another? Laura Gwilliams | New York University | @GwilliamsL
VGG19 ❖ 17 healthy adults ❖ 19-layer CNN Parallel Analysis ❖ 306 channel MEG ❖ Image Classification ❖ motor response ambiguity decision evidence Time / Layer stimulus pair (4H /6E) position (left /right) Laura Gwilliams | New York University | @GwilliamsL
Roadmap What is the order of operations performed on the sensory input? What are the underlying computations at the decision stage? How are the stages linked to one another? Laura Gwilliams | New York University | @GwilliamsL
MEG Decoding Scores Laura Gwilliams | New York University | @GwilliamsL
MEG Decoding Scores Laura Gwilliams | New York University | @GwilliamsL
MEG Decoding Scores Laura Gwilliams | New York University | @GwilliamsL
MEG DNN Decoding Scores Decoding Scores Laura Gwilliams | New York University | @GwilliamsL
Roadmap What is the order of operations performed on the sensory input? What are the underlying computations at the decision stage? How are the stages linked to one another? Laura Gwilliams | New York University | @GwilliamsL
What are the underlying computations? Categorical Percept Linear Evidence P ( letter ) P ( letter ) 0.55 P ( letter ) 0.5 Laura Gwilliams | New York University | @GwilliamsL 0.45
What are the underlying computations? 0.55 0.55 P ( letter ) P ( letter ) 0.5 0.5 0.45 0.45 4 h h h h h h h 4 h h h h h h h Categorical *** Linear *** 0.55 P ( letter ) 0.5 Laura Gwilliams | New York University | @GwilliamsL 0.45
What are the underlying computations? h linear 4 4 h H linear categorical H 4 4 Laura Gwilliams | New York University | @GwilliamsL Time (s)
Roadmap What is the order of operations performed on the sensory input? What are the underlying computations at the decision stage? How are the stages linked to one another? Laura Gwilliams | New York University | @GwilliamsL
Roadmap What is the order of operations performed on the sensory input? What are the underlying computations at the decision stage? How are the stages linked to one another? Laura Gwilliams | New York University | @GwilliamsL
Linking processing stages ❖ Human performance varies on a trial to trial basis Laura Gwilliams | New York University | @GwilliamsL
Linking processing stages ❖ Where does this variation come from — during which processing stage? ❖ Are processing delays propagated through the system? B A Architecture Delay t u o - Behaviour d t a s e R e w Fastest o l S Slowest c i f i c e p s - n o i s i e t a l u m u c c A c e p s - n o i s i Laura Gwilliams | New York University | @GwilliamsL
Linking processing stages B Generalisation 1.6 t s 1.2 Train Time (s) e t s a F 1.6 0.8 t t s s Fastest e e w w o o l l 0.4 S S 1.2 Train Time (s) 0. 0.8 Slowest Test Time 0.4 0. Laura Gwilliams | New York University | @GwilliamsL
Linking processing stages B Generalisation Alignment Latency Curve 1.6 t Decoding Accuracy s 1.2 Train Time (s) e Delay Train Time (s) t s a F 0.8 t t s s e e w w o o l l 0.4 S S Delay 0. Relative Test Time Relative Test Time Test Time Laura Gwilliams | New York University | @GwilliamsL
B Generalisation Alignment Latency Curve 1.6 Fastest Decoding Accuracy 1.2 Delay Train Time (s) Train Time (s) 0.8 Slowest 0.4 Delay 0. Relative Test Time Relative Test Time Test Time processing delay accumulates processing delay emerges C D Stim Side Stim Pair Decision Ambiguity Response .5 1.0 Normalised accuracy Fastest 0.8 .4 0.6 Slowest 0.4 .3 0.2 0. 500 ms .2 r = .03 r = .12 r = .35 r = .37 r = .66 Delay (ms) relative to mean p = .79 p = .37 p = .006 ** p = .004 ** p < .001 *** 400 200 .1 0 0. -200 slope = .001 slope = .041 slope = .123 slope = .217 slope = .416 -400 400 600 800 1000 Reaction time (ms) Laura Gwilliams | New York University | @GwilliamsL
Linking processing stages Predictions Outcome .4 B Architecture Delay s t e u o .3 - d p t a s e R e o w o l l S S .2 c i f i c e p s - n o i s i e t a l u .1 m u c c A 0. Laura Gwilliams | New York University | @GwilliamsL
Discussion ❖ Behavioural delay can be linked to a processing delay from the decision stage onwards ❖ Processing stages are sequentially linked Laura Gwilliams | New York University | @GwilliamsL
Conclusion ❖ Processing stages unfold under a sequential , hierarchical cascade ❖ A decision is formed with a bayesian- linear categorical inference -type process ❖ Each processing stage is inherently linked, such that output of the previous stage feeds to the subsequent Laura Gwilliams | New York University | @GwilliamsL
With big thanks to: @GwilliamsL Collaborator Jean-Rémi King • My supervisors, Alec Marantz and • David Poeppel , and everyone in the Neuroscience of Language Lab and Poeppel Lab ! Funding: G1001 Abu Dhabi Institute Laura Gwilliams | New York University | @GwilliamsL
Recommend
More recommend
