hierarchical reinforcement learning and human behavior
play

Hierarchical Reinforcement Learning and Human Behavior Matthew - PowerPoint PPT Presentation

Dec 08, 2023 •138 likes •845 views

Hierarchical Reinforcement Learning and Human Behavior Matthew Botvinick Princeton Neuroscience Institute and Department of Psychology Princeton University a a a * v v v Knutson et al., NeuroReport , 2001 Schultz et al.,

  1. Hierarchical Reinforcement Learning and Human Behavior Matthew Botvinick Princeton Neuroscience Institute and Department of Psychology Princeton University

  2. a a a * v v v � � �

  3. Knutson et al., NeuroReport , 2001 Schultz et al., Science , 1997

  4. Matsumoto & Hikosaka, Nature , 2007 Gehring & Willoughby, Science , 2002

  5. From Glasher, Daw, Dayan & O’Doherty, 2010 From Niv, Joel & Dayan, TICS, 2006 (artwork by B. Balleine)

  6. 0.8%

  7. The Curse of Dimensionality

  8. 15.6% The Blessing of Abstraction

  9. ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! s ( t =1) s ( t =2) s ( t =3) s ( t =4) s ( t =5) s ( t =6) Botvinick, Niv & Barto, Cognition , 2009

  10. W W S W W P G W W W W After Sutton, Precup & Singh, 1999 Botvinick, Niv & Barto, Cognition , 2009

  11. ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! s ( t =1) s ( t =2) s ( t =3) s ( t =4) s ( t =5) s ( t =6) Botvinick, Niv & Barto, Cognition , 2009

  13. Botvinick & Weinstein, Trans. Royal Society, 2014

  14. Hamilton & Grafton, J Neurosci , 2006 Humpheys & Forde, Cog. Neuropsych., 2001

  15. A C Actor Actor π ( s ) DLS Critic Critic state ( s ) state ( s ) V ( s ) VS action action R ( s ) δ DA HT+ Environment Environment B D Actor Actor 1 1 π ( s ) DLS DLS DLS o Critic Critic VS state ( s ) o state ( s ) DLPFC V ( s ) o + OFC action action R ( s ) δ HT+ DA o Environment Environment Botvinick, Niv & Barto, Cognition , 2009

  16. From Curtis & D’Esposito, TICS , 2003

  17. White & Wise, Exp Br Res, 1999

  18. Miller & Cohen, Ann. Rev. Neurosci , 2001

  19. From Badre, TICS , 2008

  20. A C Actor Actor π ( s ) DLS Critic Critic state ( s ) state ( s ) V ( s ) VS action action R ( s ) δ DA HT+ 2 2 Environment Environment B D Actor Actor π ( s ) DLS DLS DLS o Critic Critic VS state ( s ) o state ( s ) DLPFC V ( s ) o + OFC action action R ( s ) δ HT+ DA o Environment Environment Botvinick, Niv & Barto, Cognition , 2009

  21. O’Reilly & Frank, Neural Computation, 2006

  22. O’Reilly & Frank, Neural Computation, 2006 Bonini et al., J. Neurosci ., 2011

  23. A C Actor Actor π ( s ) DLS Critic Critic state ( s ) state ( s ) V ( s ) VS action action R ( s ) δ DA HT+ Environment Environment 3 3 B D Actor Actor π ( s ) DLS DLS DLS o Critic Critic VS state ( s ) o state ( s ) DLPFC V ( s ) o + OFC action action R ( s ) δ HT+ DA o Environment Environment Botvinick, Niv & Barto, Cognition , 2009

  24. Schoenbaum, et al. J Neurosci . 1999

  25. A C Actor Actor π ( s ) DLS Critic Critic state ( s ) state ( s ) V ( s ) VS action action R ( s ) δ DA HT+ Environment Environment B D Actor Actor π ( s ) DLS DLS DLS o Critic Critic VS state ( s ) o state ( s ) DLPFC V ( s ) o + OFC action action R ( s ) δ HT+ DA o Environment Environment 4 4 Botvinick, Niv & Barto, Cognition , 2009

  26. ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! s ( t =1) s ( t =2) s ( t =3) s ( t =4) s ( t =5) s ( t =6)

  27. Carlos Diuk

  28. Carlos Diuk Diuk, et al. , J Neurosci, 2013

  29. “RPE” ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! s ( t =1) s ( t =2) s ( t =3) s ( t =4) s ( t =5) s ( t =6) “PPE”

  30. Jose Fernandes Alec Solway Ribas-Fernandes et al., Neuron , 2011

  31. Jose Fernandes Alec Solway Standard RL Hierarchical RL PPE 1 RPE RPE A 0 -1 B D A C E D B C E Timestep Timestep Timestep Ribas-Fernandes et al., Neuron , 2011

  32. Jose Fernandes Alec Solway ! From Yeung, et al., 2005 Ribas-Fernandes et al., Neuron , 2011

  33. Jose Fernandes Alec Solway Ribas-Fernandes et al., Neuron , 2011

  34. Search Time A 4 Log Solution Time 3 Model Evidenc 2 1 1 100 200 Episode Botvinick, Niv & Barto, Cognition , 2009

  35. The Burden of Abstraction

  36. � 1. What should be learned? � � 2. Do people learn it? � � 3. How? �

  37. Alec Solway Carlos Diuk Solway et al., PLoS Comp. Biol., 2014

  38. Alec Solway Carlos Diuk Solway et al., PLoS Comp. Biol., 2014

  39. Alec Solway Carlos Diuk Solway et al., PLoS Comp. Biol., 2014

  40. Alec Solway Carlos Diuk !!!!!!!!!!!!!!! Pr !"#" !"#$% = Solway et al., PLoS Comp. Biol., 2014

  41. !!!!!!!!!!!!!!! Pr !"#" !"#$% = Pr !"#" !"#$% , ! Pr ! ! !"#$% , ! ∈ ! Codelength Search Time !!!!!!!!!!!!!!! Pr !"#" !"#$% = Model Evidence Solway et al., PLoS Comp. Biol., 2014

  42. Solway et al., PLoS Comp. Biol., 2014

  43. Zachary’s karate club Santa Fe Institute Lusseau’s bottlenose dolphins collaborations Fortunato, Physics Reports , 2010

  44. Simsek, Wolfe & Barto, 2005

  45. � 1. What should be learned? � � 2. Do people learn it? � � 3. How? �

  46. Carlos Diuk DebbieYee Solway et al., PLoS Comp. Biol., 2014

  47. Carlos Diuk DebbieYee Solway et al., PLoS Comp. Biol., 2014

  48. Carlos Diuk DebbieYee Solway et al., PLoS Comp. Biol., 2014

  49. Carlos Diuk DebbieYee B 2900 2800 G S ������������� 2700 2600 2500 2400 2300 2200 ���� Reject Solway et al., PLoS Comp. Biol., 2014

  50. � 1. What should be learned? � � 2. Do people learn it? � � 3. How? �

  51. , !!!!!!!!!!!!!!! Pr !"#" !"#$% = Pr !"#" !"#$% , ! Pr ! ! !"#$% ! ∈ !

  52. Anna Schapiro Schapiro et al., Nature Neurosci, 2013

  53. 1.00 0.66 -0.36 0.66 1.00 -0.36 -0.36 -0.36 1.00 Schapiro et al., Nature Neurosci, 2013

  54. Schapiro et al., Nature Neurosci, 2013

  55. Schapiro et al., Nature Neurosci, 2013

  56. Time Schapiro et al., Nature Neurosci, 2013

  57. Experiment 1 Experiment 2 All trials 0.4 0.4 Hamiltonian Probability of parse paths 0.3 0.3 0.2 0.2 0.1 0.1 0 0 Cluster transition Cluster transition Other parse Other parse parse parse Schapiro et al., Nature Neurosci, 2013

  58. + HC Schapiro et al., Nature Neurosci, 2013

  59. Carlos Diuk 1.0 Successor Representation 0.9 0.8 Correlation 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0.38 0.36 0.34 Pattern Correlation 0.32 0.30 0.28 0.26 0.24 0.22 0.20 0.18 + HC Diuk et al., in prep.

  60. Current Stimulus Next Stimulus Schapiro et al., 2013.; Rogers & McClelland, 2003

  61. Codelength Search Time !!!!!!!!!!!!!!! Pr !"#" !"#$% = Model Evidence Solway et al., PLoS Comp. Biol., 2014

  62. cf. Dayan, 1993

  63. Rosvall & Bergstrom, PNAS, 2008

  64. Mahadevan & Maggioni, 2005

  65. Stachenfeld, Botvinick & Gershman, NIPS, 2014

  66. Olshausen & Field, Nature, 1996

  67. Botvinick & Plaut, Psych Review, 2004

  68. Conclusions � • The scaling problem in RL • Hierarchy can help • Model-free versus model-based HRL • HRL in the brain • The need for good representations • Task decomposition, bottlenecks, community detection • Prospective coding and structure discovery Codelength • Hierarchy as compression

  69. Collaborators Lab Contributors Carlos Diuk (Facebook) Jose Ribas-Fernandes (U. Victoria) Andy Barto (UMass) Anna Schapiro Yael Niv (Princeton) Alec Solway (V. Tech / UCL) Tim Rogers (Wisconsin) Kim Stachenfeld Nick Turk-Browne (Princeton) Ari Weinstein Debbie Yee (Wash. U.)

Recommend

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement Learning Animesh Garg Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning Richard S. Sutton , Doina

2.18k views • 29 slides
FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu Topic: Hierarchical RL Presenter: Thophile Gaudin Why Hierarchical

921 views • 22 slides
Language as an Abstraction for Hierarchical Deep Reinforcement Learning Paper Authors: Yiding

Language as an Abstraction for Hierarchical Deep Reinforcement Learning Paper Authors: Yiding

Language as an Abstraction for Hierarchical Deep Reinforcement Learning Paper Authors: Yiding Jiang, Shixiang Gu, Kevin Murphy, Chelsea Finn Problem Overview Learning a variety of compositional , long horizon skills while being able to

364 views • 34 slides
Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
Device Placement Optimization with Reinforcement Learning A Hierarchical Model for Device

Device Placement Optimization with Reinforcement Learning A Hierarchical Model for Device

Device Placement Optimization with Reinforcement Learning A Hierarchical Model for Device Placement A. Mirhoseini, Hieu Pham, A. Goldie et al November 2019 Problem Background Tensorflow allows user to place operators on different devices to

371 views • 15 slides
Subgoals in Hierarchical Reinforcement Learning Tianren Tang Tian Tan Shangqi Guo Xiaolin Hu

Subgoals in Hierarchical Reinforcement Learning Tianren Tang Tian Tan Shangqi Guo Xiaolin Hu

Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning Tianren Tang Tian Tan Shangqi Guo Xiaolin Hu Feng Chen Background Goal-Conditional HRL High policy suffers from non-stationary problem From MARL's

330 views • 13 slides
Hierarchical Bayesian Methods for Reinforcement Learning David Wingate wingated@mit.edu Joint

Hierarchical Bayesian Methods for Reinforcement Learning David Wingate wingated@mit.edu Joint

Hierarchical Bayesian Methods for Reinforcement Learning David Wingate wingated@mit.edu Joint work with Noah Goodman, Dan Roy, Leslie Kaelbling and Joshua Tenenbaum My Research: Agents Rich sensory data Structured prior knowledge Reasonable

1.04k views • 90 slides
FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu: DeepMind Rene Bidart (rbbidart@uwaterloo.ca) CS885 June 22, 2018

653 views • 27 slides
Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak

Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak

Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine Presented by: Samuel Yigzaw 1 OUTLINE Introduction Background Main Contributions Related Work Experiments Conclusion PAGE 2

1.4k views • 26 slides
Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi,

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi,

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi, Tuan-Fang Wang, William Yang Wang Publishing year: 2018 Presenter: David Radke CS885 University of Waterloo July, 2020 Overview Problem:

493 views • 24 slides
Interactive Reinforcement Learning Human Generated Reward Presentation for Summer Camp 2015

Interactive Reinforcement Learning Human Generated Reward Presentation for Summer Camp 2015

Interactive Reinforcement Learning Human Generated Reward Presentation for Summer Camp 2015 May 25 2015 Reinforcement Learning Trial and error learning Explore and exploit Sutton and Barto 1988 Represent, predict and control

252 views • 8 slides
CS 285 Instructor: Sergey Levine UC Berkeley Todays Lecture 1. Does reinforcement learning

CS 285 Instructor: Sergey Levine UC Berkeley Todays Lecture 1. Does reinforcement learning

Reframing Control as an Inference Problem CS 285 Instructor: Sergey Levine UC Berkeley Todays Lecture 1. Does reinforcement learning and optimal control provide a reasonable model of human behavior? 2. Is there a better explanation? 3.

782 views • 45 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning Nature Vol 518, Feb 26, 2015 The Deep Mind of Demis Hassabis Backchannel / Medium.com interview with David Levy Advanced Topics:

989 views • 68 slides
DualDICE Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections Ofir

DualDICE Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections Ofir

DualDICE Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections Ofir Nachum ,* Yinlam Chow,* Bo Dai, Lihong Li Google Research *Equal contribution Reinforcement Learning Reinforcement Learning A policy acts on an

575 views • 33 slides
Understanding Human Teaching Modalities in Reinforcement Learning Environments A Preliminary

Understanding Human Teaching Modalities in Reinforcement Learning Environments A Preliminary

Understanding Human Teaching Modalities in Reinforcement Learning Environments A Preliminary Report Slides available on the Program page of the ALIHT website. W. Bradley Knox Matthew E. Taylor and Peter Stone Knowledge! Desires! Current

437 views • 33 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

682 views • 24 slides
Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides
R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec 03 2008 1 Large State Spaces h When a problem has a large state space we can not longer represent the U or Q functions as explicit tables explicit

436 views • 18 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still assume a Marko kov decision process (MDP): A se set o of st states s s S A se set o of a actions ( s (per st state) A A mo model

96 views • 5 slides
Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning<br/><br/> 4/25/19, 8*06 PM Reinforcement Learning<br/><br/> 4/25/19, 8*06 PM Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning? Spring 2019 Created:

371 views • 15 slides

More recommend