7 motor control and reinforcement learning outline
play

7. Motor Control and Reinforcement Learning Outline A. Action - PowerPoint PPT Presentation

Feb 24, 2023 •218 likes •790 views

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

  1. 7. Motor Control and Reinforcement Learning

  2. Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2

  3. Sensory-Motor Loop � Why animals have nervous systems but plants do not: animals move � a nervous system is needed to coordinate the movement of an animal’s body � movement is fundamental to understanding cognition � Perception conditions action � Action conditions perception � profound effect of action on structuring perception is often neglected 2/23/18 COSC 494/594 CCN 3

  4. Overview • • Subcortical areas: Cortical areas: basal ganglia frontal cortex o o Ø reinforcement learning Ø connections to basal ganglia & cerebellum (reward/punishment) parietal cortex o Ø connections to “what” pathway cerebellum o Ø maps sensory information to motor outputs Ø error-driven learning Ø connections to cerebellum Ø connections to “how” pathway disinhibitory output o dynamic 2/23/18 COSC 494/594 CCN 4

  5. Learning Rules Across the Brain Learning Signal Dynamics Area Reward Error Self Org Separator Integrator Attractor Primitive Basal Ganglia +++ - - - - - - ++ - - - - Cerebellum - - - +++ - - - +++ - - - - - - Advanced Hippocampus + + +++ +++ - - - +++ Neocortex ++ +++ ++ - - - +++ +++ + = has to some extent … +++ = defining characteristic – definitely has - = not likely to have … - - - = definitely does not have 2/23/18 COSC 494/594 CCN 5 (slide < O’Reilly)

  6. Primitive, Basic Learning… Learning Signal Dynamics Area Reward Error Self Org Separator Integrator Attractor Primitive Basal Ganglia +++ - - - - - - ++ - - - - Cerebellum - - - +++ - - - +++ - - - - - - • Reward & Error = most basic learning signals (self organized learning is a luxury…) • Simplest general solution to any learning problem is a lookup table = separator dynamics 2/23/18 COSC 494/594 CCN 6 (slide < O’Reilly)

  7. A. Action Selection and Reinforcement 2/23/18 COSC 494/594 CCN 7

  8. Anatomy of Basal Ganglia Lim S-J, Fiez JA and Holt LL - Lim S-J, Fiez JA and Holt LL (2014) How may the basal ganglia contribute to auditory categorization and speech perception? Front. Neurosci. 8:230. doi: 10.3389/fnins.2014.00230 http://journal.frontiersin.org/article/10.3389/fnins.2014.00230/full 2/23/18 COSC 494/594 CCN 8

  9. Basal Ganglia and Action Selection 2/23/18 COSC 494/594 CCN 9 (slide < O’Reilly)

  10. Basal Ganglia: Action Selection motor strategies future costs eye actions & plans rewards movement • Parallel circuits select motor actions and “cognitive” actions across frontal areas 2/23/18 COSC 494/594 CCN 10 (slide based on O’Reilly)

  11. Release from Inhibition 2/23/18 COSC 494/594 CCN 11 (slide < O’Reilly)

  12. Motor Loop Pathways • Direct: striatum inhibits GPi (and SNr) • Indirect: striatum inhibits GPe, which inhibits GPi (and SNr) • Hyperdirect: cortex excites STN, which diffusely excites GPi (and SNr) • GPi inhibits thalamus, which opens motor loops 2/23/18 COSC 494/594 CCN 12

  13. Basal Ganglia System • Striatum • Thalamus * § § matrix clusters (inhib.) cells fire when both: excited (cortex) direct (Go) pathway ⟞ GPi Ø Ø Ø disinhibited (GPi) indirect (NoGo) path ⟞ GPe Ø § disinhibits FC deep layers § patch clusters • Substantia nigra pars compacta (SNc) Ø to dopaminergic system • Globus pallidus, int. segment (GPi) * § releases dopamine (DA) into striatum § excites D1 receptors (Go) § tonically active § inhibits D2 receptors (NoGo) § inhibit thalamic cells • Subthalamic nucleus (STN) • Globus pallidus, ext. segment (GPe) § hyperdirect pathway § tonically active § input from cortex § inhibits corresponding GPi neurons § diffuse excitatory output to GPi § global NoGo delays decision *and substantia nigra pars reticulata (SNr) *and superior colliculus (SC) 2/23/18 COSC 494/594 CCN 13

  14. What is Dopamine Doing? 2/23/18 COSC 494/594 CCN 14

  15. Basal Ganglia Reward Learning (Frank, 2005…; O’Reilly & Frank 2006) • Feedforward, modulatory (disinhibition) on cortex/motor (same as cerebellum) • Co-opted for higher level cognitive control ⟶ PFC 2/23/18 COSC 494/594 CCN 15 (slide < O’Reilly)

  16. Basal Ganglia Architecture: Cortically-based Loops 2/23/18 COSC 494/594 CCN 16 (slide < Frank)

  17. Fronto-basal Ganglia Circuits in Motivation, Action, & Cognition 2/23/18 COSC 494/594 CCN 17 (slide < Frank)

  18. ChR2-mediated excitation of direct- and indirect-pathway MSNs in vivo drives activity in basal ganglia circuitry 2/23/18 COSC 494/594 CCN 18 AV Kravitz et al. Nature 466(7306):622-6 (2010) doi:10.1038/nature09159

  19. Human Probabilistic Reinforcement Learning Train • Patients with Parkinson’s disease (PD) are impaired in Test cognitive tasks that require learning from A (80/20) B (20/80) positive and negative A > CDEF Choose A? feedback • Likely due to depleted dopamine • But dopamine B < CDEF medication actually Avoid B? C (70/30) D (30/70) worsens performance in some cognitive tasks, despite improving it in others Frank, Seeberger & E (60/40) F (40/60) O’Reilly (2004) 2/23/18 COSC 494/594 CCN 19 (slide based on Frank)

  20. Testing the Model: Parkinson’s and Medication Effects Seniors PD OFF PD ON Probabilistic Selection Test Performance 100 90 Percent Accuracy 80 70 60 50 Frank, Seeberger & Choose A Avoid B O’Reilly (2004) Test Condition 2/23/18 COSC 494/594 CCN 20 (slide < Frank)

  21. BG Model: DA Modulates Learning from Positive/Negative Reinforcement (A) The corticostriato-thalamo-cortical loops, including the direct (Go) and indirect (NoGo) pathways of the basal ganglia. (B) The Frank (in press) neural network model of this circuit. (C) Predictions from the model for the probabilistic selection task Michael J. Frank et al. Science 2004;306:1940-1943 Published by AAAS

  22. emergent Demonstration: BG A simplified model compared to Frank, Seeberger, & O'Reilly (2004) 2/23/18 COSC 494/594 CCN 22

  23. Anatomy of BG Gating Including Subthalamic Nucleus (STN) PFC-STN provides an override mechanism 2/23/18 COSC 494/594 CCN 23 (slide < Frank)

  24. Subthalamic Nucleus: Dynamic Modulation of Decision Threshold Conflict (entropy) in choice prob ⇒ delay decision! 2/23/18 COSC 494/594 CCN 24 (slide < Frank)

  25. B. Temporal Difference Reinforcement Learning 2/23/18 COSC 494/594 CCN 25

  26. Reinforcement Learning: Dopamine Rescorla-Wagner / Delta Rule: But no CS-onset firing – need to anticipate the future! CS-onset = future reward = f 2/23/18 COSC 494/594 CCN 26 (slide < O’Reilly)

  27. Temporal Differences Learning ⟵ this is the future! 2/23/18 COSC 494/594 CCN 27 (slide < O’Reilly)

  28. Network Implementation 2/23/18 COSC 494/594 CCN 28 (slide < O’Reilly)

  29. The RL-cond Model � ExtRew: external reward r ( t ) (based on input) � TDRewPred: learns to predict reward value � minus phase = prediction V ( t ) from previous trial � plus phase = predicted V ( t +1) based on Input � TDRewInteg: Integrates ExtRew and TDRewPred � minus phase = V ( t ) from previous trial � plus phase = V ( t +1) + r ( t ) � TD: computes temporal dif. delta value ≈ dopamine signal � compute plus – minus from TDRewInteg 2/23/18 COSC 494/594 CCN 29

  30. Classical Conditioning � Forward conditioning � unconditioned stimulus (US): doesn’t depend on experience � leads to unconditioned response (UR) � preceding conditioned stimulus (CS) becomes associated with US � leads to conditioned response (CR) � Extinction � after CS established, CS is presented repeatedly without US � CR frequency falls to pre-conditioning levels � Second-order conditioning � CS1 associated with US through conditioning � CS2 associated with CS1 through conditioning, leads to CR 2/23/18 COSC 494/594 CCN 30

  31. CSC Experiment � A serial-compound stimulus has a series of distinguishable components � A complete serial-compound (CSC) stimulus has a component for every small segment of time before, during, and after the US � Richard S. Sutton & Andrew G. Barto, “Time-Derivative Models of Pavlovian Reinforcement,” Learning and Computational Neuroscience: Foundations of Adaptive Networks , M. Gabriel and J. Moore, Eds., pp. 497–537. MIT Press, 1990 � RL-cond.proj implements this form of conditioning � somewhat unrealistic, since the stimulus or some trace of it must persist until the US 2/23/18 COSC 494/594 CCN 31

  32. RL-cond.proj 2/23/18 COSC 494/594 CCN 32

  33. emergent Demonstration: RL A simplified model of temporal difference reinforcement learning 2/23/18 COSC 494/594 CCN 33

  34. Actor - Critic 2/23/18 COSC 494/594 CCN 34 (slide < O’Reilly)

  35. Opponent-Actor Learning (OpAL) � Actor has independent G and N weights � Scaled by dopamine (DA) levels during choice � Choice based on relative activation levels � Low DA: costs amplified, benefits diminished ⇒ choice 1 � High DA: benefits amplified, costs diminished ⇒ choice 3 � Moderate DA ⇒ choice 2 � Accounts for differing costs & benefits 2/23/18 COSC 494/594 CCN 35

Recommend

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides
Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
Reinforcement Learning Stephen D. Scott (Adapted from Tom Mitchells slides) 1 Outline

Reinforcement Learning Stephen D. Scott (Adapted from Tom Mitchells slides) 1 Outline

Reinforcement Learning Stephen D. Scott (Adapted from Tom Mitchells slides) 1 Outline Control learning Control policies that choose optimal actions Q learning Convergence Temporal difference learning 2 Control Learning

576 views • 20 slides
Guidelines for Action Space Definition in Reinforcement Learning-based Traffic Signal Control

Guidelines for Action Space Definition in Reinforcement Learning-based Traffic Signal Control

Guidelines for Action Space Definition in Reinforcement Learning-based Traffic Signal Control Systems Maxime Trca, Julian Garbiso, Dominique Barth October 15, 2020 Outline I - Reinforcement Learning Applied to Traffic Signal Control II -

189 views • 16 slides
Lab 11. Speed Control of a D.C. motor Motor Characterization Motor Speed Control Project

Lab 11. Speed Control of a D.C. motor Motor Characterization Motor Speed Control Project

Lab 11. Speed Control of a D.C. motor Motor Characterization Motor Speed Control Project Generate PWM waveform 1. Amplify the waveform to drive the motor 2. Measure motor speed 3. Estimate motor parameters from measured data 4. Regulate

361 views • 22 slides
Lab 11. Speed Control of a D.C. motor Motor Characterization Motor Speed Control Project

Lab 11. Speed Control of a D.C. motor Motor Characterization Motor Speed Control Project

Lab 11. Speed Control of a D.C. motor Motor Characterization Motor Speed Control Project Generate PWM waveform 1. Amplify the waveform to drive the motor 2. Measure motor speed 3. Estimate motor parameters from measured data 4. Regulate

352 views • 23 slides
Todays Outline Reinforcement Learning Dan Weld Reinforcement Learning Q-value

Todays Outline Reinforcement Learning Dan Weld Reinforcement Learning Q-value

5/7/2012 CSE 473: Artificial Intelligence Todays Outline Reinforcement Learning Dan Weld Reinforcement Learning Q-value iteration Q-learning Exploration / exploitation Linear function approximation Linear function

159 views • 12 slides
Control Regularization for Reduced Variance Reinforcement Learning Richard Cheng, Abhinav Verma,

Control Regularization for Reduced Variance Reinforcement Learning Richard Cheng, Abhinav Verma,

Control Regularization for Reduced Variance Reinforcement Learning Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri Yisong Yue, Joel W. Burdick Reinforcement Learning Reinforcement learning (RL) studies how to use data from

204 views • 9 slides
DC Motor Control To maintain constant speed Even after Motor Power is re- Motor Power duced the

DC Motor Control To maintain constant speed Even after Motor Power is re- Motor Power duced the

DC Motor Control PWM Cog DC Motor Control Input Cog Desired Speed Position Destination Count Cog Quadrature Encoder Speed Cog Proportional w Inertia DC Motor Control To maintain constant speed Even after Motor Power is re- Motor

554 views • 9 slides
Lab 8. Speed Control of a Dc motor The Motor Drive Motor Speed Control Project 1. Generate PWM

Lab 8. Speed Control of a Dc motor The Motor Drive Motor Speed Control Project 1. Generate PWM

Lab 8. Speed Control of a Dc motor The Motor Drive Motor Speed Control Project 1. Generate PWM waveform 2. Amplify the waveform to drive the motor 3. Measure tachometer signal (motor speed) 4. Find parameters of a motor model 5. Control motor

417 views • 29 slides
Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina Fragkiadaki Course Logistics Course website : all you need to

1.13k views • 88 slides
End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning

End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning

End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning Authors: Jason D. Williams and Geoffrey Zweig Speaker: Hamidreza Shahidi Outline Introduction Model description Optimizing with

388 views • 28 slides
Todays Outline Reinforcement Learning II Dan Weld Review Reinforcement Learning Review

Todays Outline Reinforcement Learning II Dan Weld Review Reinforcement Learning Review

10/26/2012 CSE 573: Artificial Intelligence Todays Outline Reinforcement Learning II Dan Weld Review Reinforcement Learning Review MDPs New MDP Algorithm: Q-value iteration Review Q-learning Large MDPs Linear function

415 views • 8 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

541 views • 15 slides
Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost

Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost

Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost using data Solid math for the tuning DNNs support approximations Safety? Explainability? 8 th of November, 2019 S. Gros SARLEM 1 / 1 Reinforcement

311 views • 9 slides
Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning: an Introduction, 2nd Edition: Chapters 6 (6.1 6.5) Outline Reinforcement Learning Reinforcement Learning: the

587 views • 27 slides
Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper by Bello, I., Pham, H., Le, Q. V., Norouzi, M., &amp; Bengio, S. (2016) Presented by Yan Shi Outline 1. Introduction 2. Background 3. Algorithms

966 views • 27 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

552 views • 35 slides
Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3. Policy Gradient and Gradient Estimators 4. Q-prop: Sample Efficient Policy Gradient and an Off-policy Critic 5. Model Based Planning in Discrete

769 views • 53 slides
Autonomous Navigation CSE 571 Inverse Optimal Control (Inverse Reinforcement Learning) Many

Autonomous Navigation CSE 571 Inverse Optimal Control (Inverse Reinforcement Learning) Many

5/25/20 Autonomous Navigation CSE 571 Inverse Optimal Control (Inverse Reinforcement Learning) Many slides by Drew Bagnell Carnegie Mellon University 1 2 Optimal Control Solution X X Learning Y Y Cost Map (Sensor Data) (Input)

489 views • 10 slides
Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent state reward action S t R t A t R t+ 1 Environment S t+ 1

1.07k views • 52 slides
Reinforcement Learning January 28, 2010 CS 886 University of Waterloo Outline Russell

Reinforcement Learning January 28, 2010 CS 886 University of Waterloo Outline Russell

Reinforcement Learning January 28, 2010 CS 886 University of Waterloo Outline Russell &amp; Norvig Sect 21.1-21.3 What is reinforcement learning Temporal-Difference learning Q-learning 2 CS886 Lecture Slides (c) 2010 K.

306 views • 28 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *,

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *,

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *, Yonathan Efroni* and Shie Mannor *equal contribution Poster #272 Action Robust Reinforcement Learning and Applications in Continuous Control Robust MDPs

502 views • 8 slides

More recommend