Using evidence accumulation to bridge the gap between neural - PowerPoint PPT Presentation

Using evidence accumulation to bridge the gap between neural networks and symbolic cognitive control Modeling Associative Recognition with large-scale Neural Networks Jelmer Borst & Terry Stewart

Theories of Associative Recognition Global Matching Encoding Matching Response (e.g., Gillund & Shiffrin, 1984; Hintzman, 1988; Murdock, 1993; Wixted & Stretch, 2004) Dual-process Encoding Familiarity Recollection Response (e.g., Diana et al., 2006; Malmberg, 2008; Rugg & Curran, 2007; Yonelinas, 2002) ACT-R Encoding Associative retrieval Response (e.g., Anderson, 2007; Anderson & Reder, 1999; Decision Schneider & Anderson, 2012)

Borst, Ghuman, & Anderson, NeuroImage, 2016 The Magnetic Fan Experiment

Sensors and sources Sensor space Source space MEG1143 − 13 x 10 6 Left Right 5 4 3 2 Activity 1 0 − 1 − 2 − 3 − 1000 − 800 − 600 − 400 − 200 0 200 Time (ms)

Sensors and sources Data - Response Generation _rh Estimated current (x 10^-11 Am) left Source space right 2.0 1.0 0.0 -800 -700 -600 -500 -400 -300 -200 -100 0 Time (ms) Data - Response Generation _lh Estimated current (x 10^-11 Am) 2.5 left right 2.0 1.5 1.0 0.5 0.0 -800 -700 -600 -500 -400 -300 -200 -100 0 Time (ms)

Associative Recognition Task Study Phase Test Phase COMFORT – MUSTARD COMFORT – MUSTARD FLAME – CAPE FLAME – DECK METAL – SPARK BERRY – CREAM EXCHANGE – HARVEST DRUNKARD - HARVEST JELLY – MOTOR METAL – MOTOR DUNGEON – GODDESS EXCHANGE – HARVEST DRUNKARD – HARVEST FINANCE – TOURIST CAPE – DECK JELLY – MOTOR …

Associative Recognition Task Study Phase Test Phase COMFORT – MUSTARD COMFORT – MUSTARD FLAME – CAPE FLAME – DECK METAL – SPARK BERRY – CREAM EXCHANGE – HARVEST DRUNKARD - HARVEST JELLY – MOTOR METAL – MOTOR DUNGEON – GODDESS EXCHANGE – HARVEST DRUNKARD – HARVEST FINANCE – TOURIST CAPE – DECK JELLY – MOTOR … Target vs Re-paired Foil vs New Foil

Associative Recognition Task Study Phase Test Phase COMFORT – MUSTARD COMFORT – MUSTARD FLAME – CAPE FLAME – DECK METAL – SPARK BERRY – CREAM EXCHANGE – HARVEST DRUNKARD - HARVEST JELLY – MOTOR METAL – MOTOR DUNGEON – GODDESS EXCHANGE – HARVEST DRUNKARD – HARVEST FINANCE – TOURIST CAPE – DECK JELLY – MOTOR … short vs long words

Associative Recognition Task Study Phase Test Phase COMFORT – MUSTARD COMFORT – MUSTARD FLAME – CAPE FLAME – DECK METAL – SPARK BERRY – CREAM EXCHANGE – HARVEST DRUNKARD - HARVEST JELLY – MOTOR METAL – MOTOR DUNGEON – GODDESS EXCHANGE – HARVEST DRUNKARD – HARVEST FINANCE – TOURIST CAPE – DECK JELLY – MOTOR … associative fan of 1 or 2

Test Phase Fixation Probe Feedback ITI COMFORT + Correct MUSTARD 500 ms (jitter) Until Response 1000 ms 500 ms Time

Behavior Response Time EEG 2000 1500 Fan 1 / Short RT (ms) 1000 Fan 1 / Long Fan 2 / Short Fan 2 / Long 500 0 Target RP Foil New Foil

Borst, Ghuman, & Anderson, NeuroImage, 2016 MEG Results Indicated by: Word length Fan Probe Response hand 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 Stimulus Response onset Time (ms)

Borst, Ghuman, & Anderson, NeuroImage, 2016 MEG Results Indicated by: Word Visual Lexical and Semantic Access length Encoding Fan Probe Response hand 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 Stimulus Response onset Time (ms)

Borst, Ghuman, & Anderson, NeuroImage, 2016 MEG Results Indicated by: Word Visual Lexical and Semantic Access length Encoding Familiarity Fan Recollection Representation Probe Response hand 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 Stimulus Response onset Time (ms)

Borst, Ghuman, & Anderson, NeuroImage, 2016 MEG Results Indicated by: Word Visual Lexical and Semantic Access length Encoding Familiarity Fan Recollection Representation Probe Decision Response hand 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 Stimulus Response onset Time (ms)

Borst, Ghuman, & Anderson, NeuroImage, 2016 MEG Results Indicated by: Word Visual Lexical and Semantic Access length Encoding Familiarity Fan Recollection Representation Probe Decision Response Response hand 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 Stimulus Response onset Time (ms)

Large-scale neural networks: Nengo

Nengo Terry Stewart

Neural ensembles Ensembles can encode represent values multiple dimensions Nengo Represents symbols Basal Ganglia as multidimensional vectors coordinate cognition But, now we have to deal with dynamics…

Motor Precentral Cognitive Control Decision Anterior Cingulate? Posterior Parietal Thalamus Representation Basal Ganglia Prefrontal Cortex Visual Visual Semantic 2 1 Scratchpad Occipital Cortex Dorsal Temporal So, what about the Declarative dynamics? Memory

New Foil

Target

Re-paired Foil

Behavior Response Time EEG Response Time Model 2000 2000 1500 1500 RT (ms) RT (ms) 1000 1000 500 500 0 0 Target RP Foil New Foil Target RP Foil New Foil Fan 1 / Short Fan 1 / Long Fan 2 / Short Fan 2 / Long

Visual Data - Visual Encoding Motor Estimated current (x 10^-11 Am) 6 Precentral short long 5 Cognitive Control Decision 4 Anterior Cingulate? 3 Posterior Parietal 2 Thalamus 1 Representation 0 Basal 0 100 200 300 400 500 600 700 800 Ganglia Time (ms) Prefrontal Cortex Visual Visual Semantic Model - Visual Encoding Scratchpad Occipital Cortex Dorsal Temporal Length Short Length Long Estimated current 8e+04 pixels 4e+04 Declarative low-dimensional Gabor Filters 0e+00 Memory representation as Tuning Curves 0 100 200 300 400 500 600 700 800 Time (ms) alien

Familiarity Motor Data - Familiarity _lh Estimated current (x 10^-11 Am) Recollection Familiarity 4 Precentral target foil Cognitive 3 Control Decision alien? Anterior Cingulate? 2 Posterior Parietal Thalamus 1 Representation Basal 0 0 100 200 300 400 500 600 700 800 Ganglia Accumulator of Time (ms) Prefrontal Cortex Summed Similarity Visual Visual Semantic Model - Familiarity Scratchpad 2.0 Occipital Cortex Dorsal Temporal PairType Target PairType RPFoil Estimated current 1.5 PairType NewFoil 1.0 0.5 Declarative Memory 0.0 0 100 200 300 400 500 600 700 800 Time (ms)

Recollection Data - Recollection _lh Motor Estimated current (x 10^-11 Am) Recollection Familiarity 2.0 fan1 Precentral fan2 1.5 Cognitive Control Decision 1.0 Anterior Cingulate? Posterior Parietal Thalamus 0.5 0.0 Representation 0 100 200 300 400 500 600 700 800 Basal Ganglia Time (ms) Prefrontal Cortex Visual Visual Semantic Model - Recollection Scratchpad Occipital Cortex Dorsal Temporal 0.8 Fan 1 Fan 2 Estimated current 0.6 0.4 0.2 Declarative Memory 0.0 0 100 200 300 400 500 600 700 800 Time (ms)

Representation Motor Data - Representation _lh Precentral Estimated current (x 10^-11 Am) Recollection Representation Cognitive fan1 1.2 Control fan2 Decision Anterior Cingulate? 0.8 Posterior Parietal Thalamus 0.4 Representation Basal Ganglia 0.0 -800 -700 -600 -500 -400 -300 -200 -100 0 Prefrontal Cortex Visual Time (ms) Visual Semantic Scratchpad Model - Representation Occipital Cortex Dorsal Temporal Fan 1 8e+05 Fan 2 Estimated current 4e+05 Declarative Memory 0e+00 -800 -700 -600 -500 -400 -300 -200 -100 0 Time (ms)

Motor Motor Precentral Cognitive Data - Response Generation _lh Data - Response Generation _rh Control Estimated current (x 10^-11 Am) Estimated current (x 10^-11 Am) Decision 2.5 left left Anterior Cingulate? right right 2.0 2.0 Posterior Parietal Thalamus 1.5 1.0 1.0 Representation Basal 0.5 Ganglia Prefrontal Cortex 0.0 0.0 Visual -800 -700 -600 -500 -400 -300 -200 -100 0 -800 -700 -600 -500 -400 -300 -200 -100 0 Visual Semantic Scratchpad Time (ms) Time (ms) Occipital Cortex Model - Left Motor Dorsal Temporal Model - Right Motor 0.8 0.8 Hand Left Hand Left Hand Right Hand Right Estimated current Estimated current 0.6 0.6 0.4 0.4 Declarative 0.2 0.2 Memory 0.0 0.0 -800 -700 -600 -500 -400 -300 -200 -100 0 -800 -700 -600 -500 -400 -300 -200 -100 0 Time (ms) Time (ms)

So, what about the dynamics? No production rules No memory retrieval that is done that are on or off

Symbolic Model ACT-R Model Production Visual Decide & Respond Retrieval Problem State Encoding Associative Retrieval Manual Familiarity Borst, Schneider, Walsh, & Anderson, JOCN, 2013 Borst & Anderson, NeuroImage, 2015 Anderson, Zhang, Borst, & Walsh, Psychological Review, 2016 Zhang, Walsh, & Anderson, JOCN, 2017

Accumulators as a solution?