Exploring Neural Mechanisms for Prediction Keith L. Downing The - PowerPoint PPT Presentation

Primitives Procedural -vs- Declarative Predictive Networks Exploring Neural Mechanisms for Prediction Keith L. Downing The Norwegian University of Science and Technology (NTNU) Trondheim, Norway keithd@idi.ntnu.no February 19, 2011 Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Philosophy: What is Prediction? Procedural -vs- Declarative Predictive Networks Neuroscience: Synaptic Modification Prediction: Essential for Action and Cognition i of the Vortex: From Neurons to Self . Llinas, 2001 You only need a brain if you move The faster and more intricate the moves, the more you need to predict their outcomes, since sensory processing is slow. On Intelligence . Hawkins, 2004 Intelligence and understanding started as a memory system that fed predictions into the sensory stream. These predictions are the essence of understanding. To know something means that you can make predictions about it... We can now see where Alan Turing went wrong. Prediction, not behavior, is the proof of intelligence. Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Philosophy: What is Prediction? Procedural -vs- Declarative Predictive Networks Neuroscience: Synaptic Modification Cognitive Incrementalism Llinas (pg. 35) ...that which we call thinking is the evolutionary internalization of movement.. Mindware (pg. 135), Andy Clark, 2001 This is the idea that you do indeed get full-blown human cognition by gradually adding bells and whistles to basic (embodied and embedded) strategies of relating to the present at hand. Is the predictive machinery evolved for motion also used for cognition? Could it be the basis of common sense ? Is it the key to Artificial General Intelligence (AGI)? Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Philosophy: What is Prediction? Procedural -vs- Declarative Predictive Networks Neuroscience: Synaptic Modification Predict To declare or indicate in advance To foretell on the basis of observation Declare To make known formally, officially, or explicitly Indicate To point out or point to To be a sign , symptom or index of ...Webster’s Online Dictionary Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Philosophy: What is Prediction? Procedural -vs- Declarative Predictive Networks Neuroscience: Synaptic Modification Declarative Prediction Recognition Associative Learning Prediction Time Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Philosophy: What is Prediction? Procedural -vs- Declarative Predictive Networks Neuroscience: Synaptic Modification Procedural Prediction To an observer, the agent’s actions indicate knowledge of a future world state. Observer Time Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Philosophy: What is Prediction? Procedural -vs- Declarative Predictive Networks Neuroscience: Synaptic Modification Eye Tracking Simulations Kettner, Mahamud, Leung, Sitkoff, Houk, Peterson and Barto (1997) This tracking behavior is considered predictive because visual signals are processed by the smooth pursuit system with considerable delays ( ≈ 100 ms)...One would expect tracking to lag by similar delays if the eye were controlled exclusively by a simple negative feedback system based on visual input...(pg. 2115) Procedurally Predictive To an observer, it may appear that the controller has an explicit representation of the ball’s future location, but actually it just knows how to move the eye to point there. Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Philosophy: What is Prediction? Procedural -vs- Declarative Predictive Networks Neuroscience: Synaptic Modification Spike-Timing Dependent Plasticity (STDP) Markram et. al. (1997), Bi et. al. (1998) ∆ W LTP - ∆ T* ∆ T* ∆ T = Tpost - Tpre LTD ∆ T* = 10 - 50 ms Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Philosophy: What is Prediction? Procedural -vs- Declarative Predictive Networks Neuroscience: Synaptic Modification Bi-Modal Thresholding Artola, Brocher and Singer (1990) Prediction Dendrites Soma N Sensory Input LTP P+S Synaptic S Strength Change 0 P- prediction only S - sensory input only LTD P+S - sensory input and prediction P Stimulation Intensity Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Philosophy: What is Prediction? Procedural -vs- Declarative Predictive Networks Neuroscience: Synaptic Modification Chemistry of Eligibility Traces Houk, Adams and Barto (1998) Neurotransmitter 2nd Messenger Promote Glutamate Dopamine Inhibit Depolarization Receptors cAMP Ca ++ Kinase A High for ≈ 100 ms LTP / LTD after depolarization Phosphatase Ca ++CAM DARPP-32 2B Phosphatase CAM PK-II 2A PO4 Autophosphorylation Disinhibition Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural Networks Procedural -vs- Declarative Predictive Networks Declarative Networks Generic Procedural Prediction Network Internally-Generated Sensory Inputs Activation Patterns Excite Inhibit Competitive Context-Detecting Layer Feedback Signals Salient Event Actions Detection Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural Networks Procedural -vs- Declarative Predictive Networks Declarative Networks The Cerebellum Parallel Fibers Granular Cells Golgi Cells Mossy Climbing Purkinje Fibers Fibers Cells Inferior Olive Sensory + Cortical Inputs Inhibition of deep cerebellar To neurons Somatosensory (touch, pain, body position) Spinal + Cortical Inputs Cord To Cerebral Efference Copy Cortex Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural Networks Procedural -vs- Declarative Predictive Networks Declarative Networks Basal Ganglia: Anatomy Cortex Striatum Caudate Nucleus Putamen VL Nucleus Globus Excite Palllidus of Thalamus (GP) Inhibit Endopedunclar Nucleus (EP) STN Substantia Nigra Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural Networks Procedural -vs- Declarative Predictive Networks Declarative Networks Basal Ganglia: Function Houk, Davis and Beiser (1998). Models of Info Proc in the BG Neocortex Thalamus Striatum Hyperdirect Pathway Indirect Direct Direct Excite Pathway Pathway Pathway Inhibit STN GP Striosome Dopamine Matriosome EP Actors SNr SNc Critic Primary reinforcement from the limbic system Midbrain & Brainstem Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural Networks Procedural -vs- Declarative Predictive Networks Declarative Networks Mixed-Temporal Context Coding Parallel Fibers Golgi Cells Granular Cells Mossy Fibers et+2 et+3 et+1 et+1 et+1 et+1 et et-2 Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural Networks Procedural -vs- Declarative Predictive Networks Declarative Networks Predictive Learning in Procedural Networks P1 Feedback Salient P2 Event Detected P3 Context Action Sensory & Internal Patterns Salient Event Feedback Salient Occurs Context Detected 100 ms Time At time T, and given situation at T − δ 1 compute the best action for T + δ 2 Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural Networks Procedural -vs- Declarative Predictive Networks Declarative Networks Eligibility Traces in the Cerebellum LTD: Those PF-PC synapses active ≈ 100 ms before error detection ( ≈ when error occured) are ⇓ most. C1 A1 Error C4 A1 Detected C1 A3 Eligibility Trace Error Occurs C2 100 ms A2 C8 A2 Time Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural Networks Procedural -vs- Declarative Predictive Networks Declarative Networks Eligibility Traces in the Basal Ganglia LTP: Corticostriatal and striatal-pallidal synapses that are active 100 ms before t reward ( ≈ t salient event ) are ⇑ most. Eligibility Substantia Nigra Trace Internal C3 A3 Reward Signal Trial 1 Salient Learn Event link to reward 100 ms Trial 2 C2 A2 C3 A3 Learn link to Salient reward Event Internal Reward Signal Keith L. Downing Time Exploring Neural Mechanisms for Prediction

Primitives Procedural Networks Procedural -vs- Declarative Predictive Networks Declarative Networks Reinforcement Learning in the Basal Ganglia Time (msec) Learning Trial X Y Z R SN X Y Z R SN Z X Y R SN Z X Y R SN Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural Networks Procedural -vs- Declarative Predictive Networks Declarative Networks Regressive Procedural Prediction Displaying predictive behavior at progressively earlier times. Time Trial 2 Trial k Trial 1 Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural Networks Procedural -vs- Declarative Predictive Networks Declarative Networks Procedural Prediction in CB and BG Neural architectures and dynamics that adapt (over 1 evolutionary and lifetime) timescales to inherent delays in sensory processing and motor activation. Implicit predictive knowledge in context-action circuitry: 2 Choose appropriate actions for time T + δ 2 based on state of the world at T − δ 1 . Learn to indicate future situations well before they occur. Keith L. Downing Exploring Neural Mechanisms for Prediction