prefrontal cortex as a meta reinforcement learning system
play

Prefrontal cortex as a Meta-reinforcement learning system Matthew - PowerPoint PPT Presentation

Aug 30, 2022 •45 likes •535 views

Prefrontal cortex as a Meta-reinforcement learning system Matthew Botvinick DeepMind, London UK Gatsby Computational Neuroscience Unit, UCL Mnih et al, Nature (2015) Mnih et al, Nature (2015) Yamins & DiCarlo, 2016 Schultz et al, Science

  1. Prefrontal cortex as a Meta-reinforcement learning system Matthew Botvinick DeepMind, London UK Gatsby Computational Neuroscience Unit, UCL

  2. Mnih et al, Nature (2015)

  4. Mnih et al, Nature (2015)

  5. Yamins & DiCarlo, 2016

  6. Schultz et al, Science (1997)

  7. Jederberg et al., 2016

  8. Jederberg et al., 2016

  9. Mante et al., Nature , 2013 Song et al., Elife , 2017

  11. Lake et al, BBS (2017)

  13. “Learning to learn” Harlow, Psychological Review, 1949

  14. “Learning to learn” Training episodes Harlow, Psychological Review, 1949

  15. Mnih et al, Nature (2015)

  17. Jederberg et al., 2016

  18. Jederberg et al., 2016

  19. https://deepmind.com/blog/impala-scalable-distributed-deeprl-dmlab-30/

  21. a v t t (PFC) δ (DA) o a r t t - 1 t - 1 Wang et al., Nature Neuroscience (2018), Wang et al., Cog. Sci., 2016; Duan et al., arXiv (2016)

  22. 0.7 0.4 0.6 0.9 0.3 0.1 0.8 0.7 Wang et al., Nature Neuroscience (2018), Wang et al., Cog. Sci. ( 2016)

  23. a v t t (PFC) δ (DA) o a r t t - 1 t - 1 Wang et al., Nature Neuroscience (2018), Wang et al., Cog. Sci. ( 2016)

  24. Left Right 4 Gittins indices Cumulative regret Thompson sampling UCB 3 Episode 2 1 1 20 40 60 80 100 Trial Trial Wang et al., Nature Neuroscience (2018), Wang et al., Cog. Sci. ( 2016)

  25. a t v t (PFC) δ (DA) o t a t-1 r t-1 Wang et al., Nature Neuroscience (2018), Wang et al., Cog. Sci. ( 2016)

  26. 0.7 0.3 0.6 0.4 0.3 0.7 0.8 0.2 Wang et al., Nature Neuroscience (2018), Wang et al., Cog. Sci. ( 2016)

  27. 4 Gittins indices Cumulative regret Thompson sampling UCB 3 Episode 2 1 1 20 40 60 80 100 Trial Trial Wang et al., Nature Neuroscience (2018), Wang et al., Cog. Sci. ( 2016)

  28. Training episodes Wang et al., Nature Neuroscience (2018), Wang et al., Cog. Sci. ( 2016)

  29. a v t t (PFC) δ (DA) o a r t t - 1 t - 1 Volkmann et al., Nature Reviews Neurology, 2010

  30. 4 4 2 2 C R C L C R C L 0 0 log 2 log 2 -2 -2 -4 -4 -4 -2 0 2 4 Tsutsui et al., Nature Comms, 2016 -4 -2 0 2 4 R R R R log 2 log 2 R L R L Wang et al., Nature Neuroscience (2018)

  31. a v t t (PFC) δ (DA) o a r t t - 1 t - 1 Wang et al., Nature Neuroscience (2018)

  32. 0.6 0.6 0.5 0.5 Correlation Proportion 0.4 0.4 0.3 0.3 Tsutsui et al., Nature Comms, 2016 0.2 0.2 0.1 0.1 a t-1 r t-1 a t-1 x r t-1 v t a t-1 r t-1 a t-1 x r t-1 v t Wang et al., Nature Neuroscience (2018)

  33. a t v t (PFC) δ (DA) o t a t-1 r t-1 Wang et al., Nature Neuroscience (2018)

  35. 4 Gittins indices Cumulative regret Thompson sampling UCB 3 Episode 2 1 1 20 40 60 80 100 Trial Trial

  36. A 1 feedback action 0.8 Reward probability 0.6 Learning rate 0.4 Inferred/decoded volatility 0.2 0 0 20 40 60 80 100 120 140 160 180 200 Step B 1 0.8 0.6 0.4 0.2 0 0 20 40 60 80 100 120 140 160 180 200 Step Behrens et al., Nature Neuroscience, 2007 Wang et al., Nature Neuroscience (2018)

  37. Behrens et al., Nature Neuroscience, 2007 Wang et al., Nature Neuroscience (2018)

  38. a v t t (PFC) δ (DA) o a r t t - 1 t - 1 Volkmann et al., Nature Reviews Neurology, 2010

  39. REVERSAL Bromberg-Martin et al, J Neurophys, 2010 Wang et al., Nature Neuroscience (2018)

  40. a v t t Left rewarded Right rewarded (PFC) δ (DA) o a r t t - 1 t - 1 Wang et al., Nature Neuroscience (2018)

  42. Model-based RL (from model-free RL) r 2 = 0.89 1 Meta-RL RPE 0 -1 Stage 2 Reward -1 0 1 Model-based RPE Wang et al., Nature Neuroscience (2018) Miller, Botvinick & Brody, Nat. Neuro., 2017; Daw et al., Neuron, 2011

  43. Optogenetic manipulation of dopamine DA blocked upon DA blocked upon DA triggered upon food reward from food reward from food omission from large/risky option small/certain option large/risky option Stopper et al., Neuron, 2014 Wang et al., arXiv; 2018

  44. Mnih et al, Nature (2015)

  47. Current / Future Work • Richer environments / abstractions (Espeholt et al., arXiv, 2018) • Architectural biases (e.g., Raposo et al., NIPS , 2017) • Complementary forms of meta-learning (e.g., Fernando et al., under review) • Episodic reinstatement (Ritter et al., in press)

  48. Neuroscience and AI: A virtuous circle

  49. Collaborators Jane Wang Adam Santoro Zeb Kurth-Nelson Tim Lillicrap Dharshan Kumaran David Barrett Chris Summerfield Dhruva Tirumala Hubert Soyer Remi Munos Joel Leibo Charles Blundell Sam Ritter Demis Hassabis DeepMind, London UK Gatsby Computational Neuroscience Unit, UCL

Recommend

Prefrontal cortex as a meta-reinforcement learning system Wang et al. CS330 Student

Prefrontal cortex as a meta-reinforcement learning system Wang et al. CS330 Student

Prefrontal cortex as a meta-reinforcement learning system Wang et al. CS330 Student Presentation Motivation Computational Neuro: AI <> Neurobio Feedback Loop Convolutions and the eye, SNNs and Learning Rules, etc. Meta

283 views • 16 slides
Meta Reinforcement Learning as Task Inference Jan Humplik, Alexandre Galashov, Leonard

Meta Reinforcement Learning as Task Inference Jan Humplik, Alexandre Galashov, Leonard

Meta Reinforcement Learning as Task Inference Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A.Ortega, Yee Whye Teh, Nicholas Heess Topic: Bayesian RL Presenter: Ram Ananth Why meta Reinforcement Learning? First Wave of Deep

912 views • 63 slides
ADDICTION Prefrontal Cortex Affect Dysregulation Window of Tolerance Internal

ADDICTION Prefrontal Cortex Affect Dysregulation Window of Tolerance Internal

NEUROBIOLOGY OF ADDICTION Prefrontal Cortex Affect Dysregulation Window of Tolerance Internal Working Model Developing Brain Shame psychologically =Unseen Neurologically = Ventral Tegmental In high

857 views • 44 slides
Journal- prefrontal cortex hypoactivity prevents compulsive cocaine Published Weekly (51

Journal- prefrontal cortex hypoactivity prevents compulsive cocaine Published Weekly (51

Rescuing cocaine-induced Journal- prefrontal cortex hypoactivity prevents compulsive cocaine Published Weekly (51 issues) seeking Founded 1869- England Impact factor 38.597 BIONB 4110 Multidisciplinary March 10 th ,

168 views • 5 slides
Meta Reinforcement Learning Kate Rakelly 11/13/19 Questions we seek to answer Motivation : What

Meta Reinforcement Learning Kate Rakelly 11/13/19 Questions we seek to answer Motivation : What

Meta Reinforcement Learning Kate Rakelly 11/13/19 Questions we seek to answer Motivation : What problem is meta-RL trying to solve? Context : What is the connection to other problems in RL? Solutions : What are solution methods for meta-RL and

787 views • 52 slides
Reinforcement Learning by the People and for the People: With a Focus on Lifelong / Meta /

Reinforcement Learning by the People and for the People: With a Focus on Lifelong / Meta /

Reinforcement Learning by the People and for the People: With a Focus on Lifelong / Meta / Transfer Learning Emma Brunskill Stanford CS234 Winter 2018 Quiz Information Monday, in class See piazza for room information (Released by

942 views • 81 slides
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning Kelvin Xu, Ellis Ratner, Anca

Learning a Prior over Intent via Meta-Inverse Reinforcement Learning Kelvin Xu, Ellis Ratner, Anca

Learning a Prior over Intent via Meta-Inverse Reinforcement Learning Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, Chelsea Finn University of California, Berkeley Motivation : a well specified reward function remains an important

924 views • 42 slides
Distributed Meta Optimization of Reinforcement Learning Agents Greg Heinrich, Iuri Frosio - GTC

Distributed Meta Optimization of Reinforcement Learning Agents Greg Heinrich, Iuri Frosio - GTC

Distributed Meta Optimization of Reinforcement Learning Agents Greg Heinrich, Iuri Frosio - GTC San Jose, March 2019 Contents Introduction to Reinforcement Learning Introduction to Metaoptimization (on distributed systems) / Maglev

596 views • 45 slides
Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly, K., Zhou, A., Quillen, D., Finn, C., & Levine, S ICML, 2019 Presented by: Egill Ian Gudmundsson Some Terminology On-policy learning: Only one

427 views • 16 slides
PEARL Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables Kate

PEARL Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables Kate

PEARL Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables Kate Rakelly*, Aurick Zhou*, Deirdre Quillen, Chelsea Finn, Sergey Levine Hula Beach, Never grow up, The Sled - by artist Matt Spangler,

603 views • 12 slides
Unsupervised Meta-Learning for Reinforcement Learning LAMDA, Nanjing University . .

Unsupervised Meta-Learning for Reinforcement Learning LAMDA, Nanjing University . .

. . . . . . . . . . . . . . . . . Unsupervised Meta-Learning for Reinforcement Learning LAMDA, Nanjing University . . . . . . . . . . . . . . . . . . . . . . . November 9, 2020 . . . . . .

664 views • 47 slides
Hippocampal-prefrontal plasticity seems to reverberate in a thalamic-prefrontal loop: what else

Hippocampal-prefrontal plasticity seems to reverberate in a thalamic-prefrontal loop: what else

Hippocampal-prefrontal plasticity seems to reverberate in a thalamic-prefrontal loop: what else neuromathematics could tell us? Lzio S. Bueno-Jnior Joo P. Leite Medical School of Ribeiro Preto 1/5 First, the structure: a particular

487 views • 23 slides
Causal Reasoning from Meta-reinforcement Learning Dasgupta et al. (2018) CS330 Student

Causal Reasoning from Meta-reinforcement Learning Dasgupta et al. (2018) CS330 Student

Causal Reasoning from Meta-reinforcement Learning Dasgupta et al. (2018) CS330 Student Presentation Background: Why Causal Reasoning? There is only so much of the world we can understand via observation. Cancer (correlates to) Smoking

297 views • 19 slides
Meta Reinforcement Learning Chelsea Finn Why are humans so good at RL? People have prior

Meta Reinforcement Learning Chelsea Finn Why are humans so good at RL? People have prior

Meta Reinforcement Learning Chelsea Finn Why are humans so good at RL? People have prior experience. People have an existing representation of the world. Can we learn a representation under which RL is fast? Key idea : Explicitly optimize for

1.04k views • 41 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885

Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885

Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885 Reinforcement Learning Paper by: P. Zhang, L. Xiong, Z. Yu, P. Fang, S. Yan, J. Yao, and Y. Zhou (Sensors 2019) Presented by: Neel Bhatt Context and

286 views • 14 slides
Paths to Learning Reinforcement Learning Ever wonder how the orcas and dolphins at Sea World are

Paths to Learning Reinforcement Learning Ever wonder how the orcas and dolphins at Sea World are

Paths to Learning Reinforcement Learning Ever wonder how the orcas and dolphins at Sea World are trained? They learn through a simple system of reinforcement learning, where the desired behaviors are met with rewards such as raw fish and back

248 views • 5 slides
Meta Learning Shengchao Liu Background Meta Learning (AKA Learning to Learn) A

Meta Learning Shengchao Liu Background Meta Learning (AKA Learning to Learn) A

Meta Learning Shengchao Liu Background Meta Learning (AKA Learning to Learn) A fast-learning algorithm: quickly adapted from the source tasks to the target tasks Key terminologies Support Set & Query Set C-Way K-Shot

2k views • 41 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
Symbolic Regression for Reinforcement Learning and Dynamic System Modeling Robert Babuka 1

Symbolic Regression for Reinforcement Learning and Dynamic System Modeling Robert Babuka 1

Symbolic Regression for Reinforcement Learning and Dynamic System Modeling Robert Babuka 1 Research interests Clustering for building locally linear models Reinforcement learning for continuous dynamic systems Neural

604 views • 31 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

683 views • 24 slides
Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides

More recommend