survey leveraging human guidance for deep reinforcement
play

Survey: Leveraging Human Guidance for Deep Reinforcement Learning - PowerPoint PPT Presentation

Jun 19, 2023 •45 likes •465 views

Survey: Leveraging Human Guidance for Deep Reinforcement Learning Tasks Ruohan Zhang, Faraz Torabi, Lin Guan, Dana H. Ballard, Peter Stone University of Texas at Austin Presented by Lin Guan Lin Guan (UT Austin) Paper#10921 1 / 33 A

  1. Survey: Leveraging Human Guidance for Deep Reinforcement Learning Tasks Ruohan Zhang, Faraz Torabi, Lin Guan, Dana H. Ballard, Peter Stone University of Texas at Austin Presented by Lin Guan Lin Guan (UT Austin) Paper#10921 1 / 33

  2. A Reinforcement Learning Problem: Montezuma’s Revenge Lin Guan (UT Austin) Paper#10921 2 / 33

  3. A Reinforcement Learning Problem: Montezuma’s Revenge Lin Guan (UT Austin) Paper#10921 3 / 33

  4. A Reinforcement Learning Problem: Montezuma’s Revenge Lin Guan (UT Austin) Paper#10921 4 / 33

  5. Learning Objective Find an optimal policy , i.e., the action to take in an observed state that maximizes expected longterm reward Lin Guan (UT Austin) Paper#10921 5 / 33

  6. Montezuma’s Revenge: Imitation Learning Lin Guan (UT Austin) Paper#10921 6 / 33

  7. Survey Scope 64 papers, 5 types of human guidance that... Lin Guan (UT Austin) Paper#10921 7 / 33

  8. Survey Scope 64 papers, 5 types of human guidance that... Are beyond conventional step-by-step action demonstrations Lin Guan (UT Austin) Paper#10921 7 / 33

  9. Survey Scope 64 papers, 5 types of human guidance that... Are beyond conventional step-by-step action demonstrations Have shown promising results in training agents to solve deep reinforcement learning tasks Lin Guan (UT Austin) Paper#10921 7 / 33

  10. Introduction 1 Learning from Human Evaluative Feedback 2 Learning from Human Preference 3 Hierarchical Imitation 4 Imitation from Observation 5 Learning Attention from Human 6 Conclusion 7 Lin Guan (UT Austin) Paper#10921 8 / 33

  11. Montezuma’s Revenge: Evaluative Feedback Lin Guan (UT Austin) Paper#10921 9 / 33

  12. Motivation While the true reward is delayed and sparse, human evaluative feedback is immediate and dense. Lin Guan (UT Austin) Paper#10921 10 / 33

  13. Representative Works Interpreting human feedback as: Reward function, replacing reward provided by the environment TAMER: Training an agent manually via evaluative reinforcement [Knox and Stone, 2009, Warnell et al., 2018] Lin Guan (UT Austin) Paper#10921 11 / 33

  14. Representative Works Interpreting human feedback as: Direct policy labels Advise [Griffith et al., 2013, Cederborg et al., 2015] Lin Guan (UT Austin) Paper#10921 12 / 33

  15. Representative Works Interpreting human feedback as: Direct policy labels Advise [Griffith et al., 2013, Cederborg et al., 2015] Advantage function COACH: Convergent actor-critic by humans [MacGlashan et al., 2017] This interpretation explains human feedback behaviors better in several tasks Still an unresolved issue that requires carefully designed human studies Lin Guan (UT Austin) Paper#10921 12 / 33

  16. Introduction 1 Learning from Human Evaluative Feedback 2 Learning from Human Preference 3 Hierarchical Imitation 4 Imitation from Observation 5 Learning Attention from Human 6 Conclusion 7 Lin Guan (UT Austin) Paper#10921 13 / 33

  17. Montezuma’s Revenge: Human Preference Lin Guan (UT Austin) Paper#10921 14 / 33

  18. Motivation Ranking behaviors is easier than rating them. And sometimes the ranking can only be provided at the end of a behavior trajectory. Lin Guan (UT Austin) Paper#10921 15 / 33

  19. Representative Works [Christiano et al., 2017]: As an inverse reinforcement learning problem, i.e., learn human reward function from human preference rather than from demonstration Lin Guan (UT Austin) Paper#10921 16 / 33

  20. Representative Works [Christiano et al., 2017]: As an inverse reinforcement learning problem, i.e., learn human reward function from human preference rather than from demonstration Query selection? Preference elicitation [Zintgraf et al., 2018] Lin Guan (UT Austin) Paper#10921 16 / 33

  21. Representative Works [Christiano et al., 2017]: As an inverse reinforcement learning problem, i.e., learn human reward function from human preference rather than from demonstration Query selection? Preference elicitation [Zintgraf et al., 2018] Many good works on preference-based reinforcement learning [Wirth et al., 2017] Lin Guan (UT Austin) Paper#10921 16 / 33

  22. Introduction 1 Learning from Human Evaluative Feedback 2 Learning from Human Preference 3 Hierarchical Imitation 4 Imitation from Observation 5 Learning Attention from Human 6 Conclusion 7 Lin Guan (UT Austin) Paper#10921 17 / 33

  23. Montezuma’s Revenge: Hierarchical Imitation Lin Guan (UT Austin) Paper#10921 18 / 33

  24. Motivation Human is good at specifying high-level abstract goals while the agent is good at performing low-level fine-grained controls. Lin Guan (UT Austin) Paper#10921 19 / 33

  25. Representative Works High-level+low-level demonstrations [Le et al., 2018] Lin Guan (UT Austin) Paper#10921 20 / 33

  26. Representative Works High-level+low-level demonstrations [Le et al., 2018] High-level demonstrations only [Andreas et al., 2017] Lin Guan (UT Austin) Paper#10921 20 / 33

  27. Representative Works High-level+low-level demonstrations [Le et al., 2018] High-level demonstrations only [Andreas et al., 2017] A promising combination: High-level: Imitation learning, e.g., DAgger [Ross et al., 2011] Low-level: Reinforcement learning, e.g., DQN [Mnih et al., 2015] Lin Guan (UT Austin) Paper#10921 20 / 33

  28. Introduction 1 Learning from Human Evaluative Feedback 2 Learning from Human Preference 3 Hierarchical Imitation 4 Imitation from Observation 5 Learning Attention from Human 6 Conclusion 7 Lin Guan (UT Austin) Paper#10921 21 / 33

  29. Montezuma’s Revenge: Imitation from Observation Lin Guan (UT Austin) Paper#10921 22 / 33

  30. Motivation To utilize a large amount of human demonstration data that do not have action labels, e.g., YouTube videos Lin Guan (UT Austin) Paper#10921 23 / 33

  31. Representative Works Challenge 1: Perception Viewpoint [Liu et al., 2018, Stadie et al., 2017] Embodiment [Gupta et al., 2018, Sermanet et al., 2018] Lin Guan (UT Austin) Paper#10921 24 / 33

  32. Representative Works Challenge 1: Perception Viewpoint [Liu et al., 2018, Stadie et al., 2017] Embodiment [Gupta et al., 2018, Sermanet et al., 2018] Challenge 2: Control Model-based: Infer the missing action given a state transitions ( s , s ′ ) by learning an inverse dynamics model [Nair et al., 2017, Torabi et al., 2018a] Model-free: e.g., bring the state distribution of the imitator closer to that of the trainer using generative adversarial learning [Merel et al., 2017, Torabi et al., 2018b] Lin Guan (UT Austin) Paper#10921 24 / 33

  33. Representative Works Challenge 1: Perception Viewpoint [Liu et al., 2018, Stadie et al., 2017] Embodiment [Gupta et al., 2018, Sermanet et al., 2018] Challenge 2: Control Model-based: Infer the missing action given a state transitions ( s , s ′ ) by learning an inverse dynamics model [Nair et al., 2017, Torabi et al., 2018a] Model-free: e.g., bring the state distribution of the imitator closer to that of the trainer using generative adversarial learning [Merel et al., 2017, Torabi et al., 2018b] Please see paper#10945: Recent Advances in Imitation Learning from Observation [Torabi et al., 2019] Lin Guan (UT Austin) Paper#10921 24 / 33

  34. Introduction 1 Learning from Human Evaluative Feedback 2 Learning from Human Preference 3 Hierarchical Imitation 4 Imitation from Observation 5 Learning Attention from Human 6 Conclusion 7 Lin Guan (UT Austin) Paper#10921 25 / 33

  35. Montezuma’s Revenge: Human Attention Lin Guan (UT Austin) Paper#10921 26 / 33

  36. Motivation Human visual attention provides additional information on why a particular decision is made, e.g., by indicating the current object of interest. Lin Guan (UT Austin) Paper#10921 27 / 33

  37. Representative Works AGIL: Attention-guided imitation learning [Zhang et al., 2018] Including attention does lead to higher accuracy in imitating human actions Lin Guan (UT Austin) Paper#10921 28 / 33

  38. Representative Works (a) Cooking [Li et al., 2018] (b) Driving [Palazzi et al., 2018, Xia et al., 2019] Lin Guan (UT Austin) Paper#10921 29 / 33

  39. Survey Scope An agent can learn... From human evaluative feedback From human preference From high-level goals specified by humans By observing human performing the task From human visual attention Lin Guan (UT Austin) Paper#10921 30 / 33

  40. Future Directions Shared datasets and reproducibility Understanding human trainers’ behaviors, e.g.,[Thomaz and Breazeal, 2008] A unified lifelong learning framework [Abel et al., 2017] Lin Guan (UT Austin) Paper#10921 31 / 33

  41. Survey: Leveraging Human Guidance for Deep Reinforcement Learning Tasks Ruohan Zhang, Faraz Torabi, Lin Guan, Dana H. Ballard, Peter Stone University of Texas at Austin Presented by Lin Guan Thank You! Lin Guan (UT Austin) Paper#10921 32 / 33

Recommend

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning Nature Vol 518, Feb 26, 2015 The Deep Mind of Demis Hassabis Backchannel / Medium.com interview with David Levy Advanced Topics:

989 views • 68 slides
SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background

SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background

SDRL: Symbolic Deep Reinforcement Learning SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background Leveraging Symbolic Planning Method Experiment Conclusion Bo Liu and Future Work Auburn

1k views • 20 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Human-level control through deep reinforcement learning Volodymyr Mnih, Koray Kavukcuoglu, David

Human-level control through deep reinforcement learning Volodymyr Mnih, Koray Kavukcuoglu, David

Human-level control through deep reinforcement learning Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles

515 views • 39 slides
Human-level control through deep reinforcement Liia Butler But first... A quote "The

Human-level control through deep reinforcement Liia Butler But first... A quote "The

Human-level control through deep reinforcement Liia Butler But first... A quote "The question of whether machines can think... is about as relevant as the question of whether submarines can swim" Edsger W. Dijkstra Overview 1.

338 views • 21 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides
Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics: Frontiers and Beyond Shixiang (Shane) Gu ( ) 2018.5.27 01 Deep RL: successes and limitations Computation-Constrained Data-Constrained

634 views • 21 slides
Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Introduction to Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 1 is due next Wednesday! Remember that Monday is a holiday, so no office hours 2. Remember to start forming final project

877 views • 48 slides
Assessing Generalization in Deep Reinforcement Learning Soo Jung Jang Background Before (ex:

Assessing Generalization in Deep Reinforcement Learning Soo Jung Jang Background Before (ex:

Assessing Generalization in Deep Reinforcement Learning Soo Jung Jang Background Before (ex: factory robot) Now (ex: human-like intelligence) focus on one environment apply to multiple environment generalization is not considered

142 views • 12 slides
Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 is extended by one week, to Wednesday after next Todays Lecture 1. Managing overfitting in model-based RL

686 views • 40 slides
Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina Fragkiadaki Course Logistics Course website : all you need to

1.13k views • 88 slides
Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides
Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning for robots? slide from V. Vanhoucke End-to-end Deep Learning for robots? 2010 : Speech Recognition Audio Acoustic Model Phonetic Model

1.05k views • 65 slides
CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

DeepLoco by X. B. Peng, G. Berseth & M. van de Panne CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe University // Spring 2018 Neural Face by Taehoon Kim Previously on CMP784 Generative

895 views • 71 slides
Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3. Policy Gradient and Gradient Estimators 4. Q-prop: Sample Efficient Policy Gradient and an Off-policy Critic 5. Model Based Planning in Discrete

769 views • 53 slides
Cooperation and Communication in Multiagent Deep Reinforcement Learning Matthew Hausknecht Nov

Cooperation and Communication in Multiagent Deep Reinforcement Learning Matthew Hausknecht Nov

Cooperation and Communication in Multiagent Deep Reinforcement Learning Matthew Hausknecht Nov 28, 2016 Advisor: Peter Stone 1 Motivation Intelligent decision making is at the heart of AI. 2 Thesis Question How can the power of Deep

1.81k views • 121 slides
10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell

10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell

10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell October 29, 2018 Reading: Barto & Sutton Chapter 15 Tom Mitchell, October 2018 Outline RL in primates RL in humans Error signals and

448 views • 20 slides
Deep Reinforcement Learning Philipp Koehn 18 April 2019 Philipp Koehn Artificial Intelligence:

Deep Reinforcement Learning Philipp Koehn 18 April 2019 Philipp Koehn Artificial Intelligence:

Deep Reinforcement Learning Philipp Koehn 18 April 2019 Philipp Koehn Artificial Intelligence: Deep Reinforcement Learning 18 April 2019 Reinforcement Learning 1 Sequence of actions moves in chess driving controls in car

859 views • 63 slides
Deep Reinforcement Learning Philipp Koehn 21 April 2020 Philipp Koehn Artificial Intelligence:

Deep Reinforcement Learning Philipp Koehn 21 April 2020 Philipp Koehn Artificial Intelligence:

Deep Reinforcement Learning Philipp Koehn 21 April 2020 Philipp Koehn Artificial Intelligence: Deep Reinforcement Learning 21 April 2020 Reinforcement Learning 1 Sequence of actions moves in chess driving controls in car

815 views • 63 slides
Deep Reinforcement Learning Lecture 1 Sergey Levine How do we build intelligent machines?

Deep Reinforcement Learning Lecture 1 Sergey Levine How do we build intelligent machines?

Deep Reinforcement Learning Lecture 1 Sergey Levine How do we build intelligent machines? Intelligent machines must be able to adapt Deep learning helps us handle unstructured environments Reinforcement learning provides a formalism for

1.89k views • 172 slides

More recommend