deep reinforcement learning for robotics frontiers and
play

Deep Reinforcement Learning for Robotics: - PowerPoint PPT Presentation

Aug 21, 2022 •415 likes •634 views

Deep Reinforcement Learning for Robotics: Frontiers and Beyond Shixiang (Shane) Gu ( ) 2018.5.27 01 Deep RL: successes and limitations Computation-Constrained Data-Constrained

  1. 深度强化学习与机器塀⼈亻：前沿与未来 Deep Reinforcement Learning for Robotics: Frontiers and Beyond Shixiang (Shane) Gu ( 顾世翔 ) 2018.5.27 01

  2. Deep RL: successes and limitations Computation-Constrained Data-Constrained Simulation = success Real-world = not applied… Atari games [Mnih et. al., 2015] ? AlphaGo/AlphaZero [Silver et. al., 2016; 2017] Parkour [Heess et. al., 2017]

  3. Why Robotics? vs

  4. ⾃臫动化 算法 Recipe for a Good Deep RL Algorithm Interpretability 可解释性 Reliability Risk-Aware ⻛飏险意识 可靠性 Transferability/Generalization 可转移性，普及性 State/Temporal Abstraction 时空抽象化 Exploration 探索 Reset-free ⽆旡重置 Universal Reward 万能奖励 Automation Human-free Learning ⽆旡需⼈亻的学习 Scalability 可扩展性 Algorithm Stability 稳定性 Sample-e ffi ciency 采样效率

  5. Outline of the talk Sample-e ffi ciency 采样效率 • Good Off-policy Algorithm 好的离策算法： NAF [Gu et al, 2016] , Q-Prop/IPG [Gu et al, 2017/2017] • Good Model-based Algorithm 好的有模型算法： TDM [Pong*, Gu* et al, 2018] • Human-free Learning ⽆旡需⼈亻的学习 • Safe & reset-free RL 安全的，⽆旡重制的强化学习： LNT [Eysenbach, Gu et al, 2018] • “Universal” reward function 万能奖励函数： TDM [Pong*, Gu* et al, 2018] • Temporal Abstraction 时间抽象化 • Data-efficient hierarchical RL ⾼髙采样效率，分层型强化学习： HIRO [Nachum, Gu et al, 2018] •

  6. Notations & Definitions on-policy model-free 在策⽆旡模型法 : e.g. policy search ～ trial and error 试错 o ff -policy model-free 离策⽆旡模型法 : e.g. Q-learning ～ introspection 反思 model-based 有模型法 : e.g. MPC ～ imagination 想象

  7. Sample-efficiency & RL controversy “ 蛋糕上的樱桃 ” Model-based On-policy Off-policy sample-e ffi ciency 采样效率 learning signals 学习信号 Less More instability 不泌稳定性

  8. Toward Good Off-policy Deep RL Algorithm On-policy Monte Carlo policy gradient, Trial & error 试错 e.g. TRPO [Schulman et al, 2015] ? • Many new samples needed per update. • Stable but very sample-intensive Actor ⾏行降动者 Critic 批评者 Off-policy actor-critic, e.g. DDPG [Lillicrap et al, 2016] • No new samples needed per Introspection 反思 update! • Quite sensitive to hyper-parameters imperfect 不泌是全知的。 “Better” DDPG • NAF [Gu et al 2016] , Double DQN [Hasselt et al 2016] , Dueling DQN [Wang et al 2016] , Q-Prop/IPG [Gu et al 2017/2017] , ICNN [Amos et al 2017] , SQL/SAC [Haarnoja et al 2017/2017] , GAC [Tangkaratt et al 2018] , MPO [Abdolmaleki et al 2018] , TD3 [Fujimoto et al 2018] , …

  9. Normalized Advantage Functions (NAF) • Benefit: 2 objectives (actor-critic) to 1 objective (Q-learning) • Halve #hyperparameters JACO arm grasp & reach • Limitation: expressibility of Q-function • Doesn’t work well on locomotion 3-joint peg insertion • Works well on manipulation Related (later) work: Dueling Network [Wang et al 2016] • ICNN [Amos et al 2017] • SQL [Hajaorna et al 2017] • [Gu, Lillicrap, Sutskever, Levine, ICML 2016]

  10. Asynchronous NAF for Simple Manipulation Train time/Exploration Test time Disturbance test 2.5 hours [Gu*, Holly*, Lillicrap, Levine, ICRA 2017]

  11. Add one eq balancing on-policy and o ff -policy grad Q-Prop & Interpolated Policy Gradient (IPG) • On-policy algorithms are stable. How to make off-policy more on-policy? • Mixing Monte Carlo returns • Trust-region policy update • On-policy exploration • Bias trade-offs (theoretically bounded) Trial & error 试错 Critic 批评者 + Related concurrent work: [Gu, Lillicrap, Ghahramani, Turner, Levine, ICLR 2017] PGQ [O’Donoghue et al 2017] • [Gu, Lillicrap, Ghahramani, Turner, Schoelkopf, Levine, NIPS 2017] ACER [Wang et al 2017] •

  12. 反思 Toward Good Model-based Deep RL Algorithm • Rethinking Q-learning • Q-learning vs parameterized Q-learning 反思 + ⽆旡限记忆改写 Off-policy + Relabeling trick from HER [Andrychowicz et al, 2017] Examples: • UVF [Schaul et al, 2015] • TDM [Pong*, Gu* et al 2017] Introspection (off-policy model-free) + relabeling = imagination (model-based)? 反思（离策⽆旡模型） + ⽆旡限记忆改写 = 想象（有模型）？

  13. Temporal Difference Models (TDM) • A certain parameterized Q-function is a generalization of dynamics model • Efficient learning by relabeling • Novel model-based planning [Pong*, Gu*, Dalal, Levine, ICLR 2018]

  14. Toward Human-free Learning ⽆旡需⼈亻的学习 ? Human-administered, Autonomous, Continual, Manual resetting ， Safe, Human-free Reward engineering

  15. 不泌落痕迹 能去能回 Leave No Trace (LNT) Who resets the robot? - PhD students • Learn to reset • Early abort based on how likely you can go back to initial state (reset Q-function) • Goal: reduce/eliminate manual resets = safe, autonomous, continual learning + curriculum Related work: Asymmetric self-play [Sukhbaatar et al 2017] • Automatic goal generation [Held et al 2017] • Reverse curriculum [Florensa et al 2017] • [Eysenbach, Gu, Ibarz, Levine, ICLR 2018]

  16. A “Universal” Reward Function “ 万能 ” 奖励函数 + Off-Policy Learning • Goal: learn as many useful skills as possible sample-e ffi ciently with minimal reward engineering • Examples: Diversity reward, e.g. Goal-reaching reward, e.g. SNN4HRL [Florensa et al 2017], DIAYN [Eysenbach et al 2018] UVF [Schaul et al 2015]/HER[Andrychowicz], TDM [Pong*, Gu* et al 2018]

  17. 时间抽象化 Toward Temporal Abstractions When you don’t know how to ride bike… When you know how to ride bike… ? TDM learns many skills very quickly… How to e ffi ciently solve other problems? ?

  18. HIerarchical Reinforcement learning with Off-policy correction (HIRO) • Most recent HRL work is on-policy • e.g. option-critic [Bacon et al 2015] , FuN [Vezhnevets et al 2017] , SNN4HRL [Florensa et al 2017] , MLSH [Frans et al 2018] • VERY data-intensive • How to correct for off-policy? Relabel the action. • 不泌是记忆改写，是 记忆纠正 [Nachum, Gu, Lee, Levine, preprint 2018]

  19. HIRO (cont.) Ant Maze Ant Push Ant Fall [Vezhnevets et al, 2017] [Florensa et al, 2017] [Houthooft et al, 2016] [Nachum, Gu, Lee, Levine, preprint 2018] Test rewards at 20000 episodes

  20. Discussion • Optimizing for computation alone is not enough; also for sample-e ffi ciency and stability ; data is valuable . • Efficient algorithms ⾼髙采样效率算法 • Human-free learning ⽆旡需⼈亻的学习 • Reliability 可靠性 + Simulation Natural language, Causality + Multi-task Distributional, Bayesian + Imitation Sim2Real, Meta learning + Human-feedback HIRO + etc. LNT/TDM + etc. NAF/Q-Prop/IPG/TDM + etc.

  21. Thank you! Contact: sg717@cam.ac.uk, shanegu@google.com Timothy Lillicrap Richard E. Turner, Zoubin Ghahramani Sergey Levine, Vitchyr Pong Bernhard Schoelkopf Ilya Sutskever (now at OpenAI), Ethan Holly, Ben Eysenbach, Ofir Nachum, Honglak Lee …and other amazing colleagues from: Cambridge, MPI Tuebingen, Google Brain, and DeepMind

Recommend

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 4: Q-Value based RL Animesh

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 4: Q-Value based RL Animesh

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 4: Q-Value based RL Animesh Garg Deep Reinforcement Learning with Double Q-learning Hado van Hasselt, Arthur Guez, David Silver Dueling Network Architectures for Deep

535 views • 24 slides
CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement Learning Animesh Garg Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning Richard S. Sutton , Doina

2.18k views • 29 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

541 views • 15 slides
1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides
Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen, Steven Bohez, Elias De

Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen, Steven Bohez, Elias De

Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen, Steven Bohez, Elias De Coninck, Sam Leroux, Pieter Van Molle Bert VanKeirsbilck, Pieter Simoens, Bart Dhoedt sam.leroux@ugent.be PUBLIC How can we build robots that are able to

705 views • 31 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Target-driven Visual Navigation in Indoor Scenes Using Deep Reinforcement Learning [Zhu et al.

Target-driven Visual Navigation in Indoor Scenes Using Deep Reinforcement Learning [Zhu et al.

REINFORCEMENT LEARNING FOR ROBOTICS Target-driven Visual Navigation in Indoor Scenes Using Deep Reinforcement Learning [Zhu et al. 2017] A. James E. Cagalawan james.cagalawan@gmail.com University of Waterloo June 27, 2018 (Note: Videos in

793 views • 45 slides
CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised & Imitation

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised & Imitation

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised & Imitation Learning Instructor: Animesh Garg TA: Dylan Turpin & Tingwu Wang Agenda Invitation to Imitation DAGGER: Dataset Aggregation

904 views • 41 slides
Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides
Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Introduction to Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 1 is due next Wednesday! Remember that Monday is a holiday, so no office hours 2. Remember to start forming final project

880 views • 48 slides
CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Behavioral Cloning from

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Behavioral Cloning from

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Behavioral Cloning from Observation Tingwu Wang, Dylan Turpin, Animesh Garg Agenda Background Problem Setting Behavior Cloning / Dagger Generative Adversarial

411 views • 12 slides
Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 is extended by one week, to Wednesday after next Todays Lecture 1. Managing overfitting in model-based RL

686 views • 40 slides
CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 1: Introduction &

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 1: Introduction &

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 1: Introduction & Logistics Animesh Garg Agenda Logistics Course Motivation Primer in RL Human learning and RL (sample paper presentation) Presentation

1.46k views • 110 slides
Soft Actor-Critic: Deep Reinforcement Learning for Robotics Finn Rietz University of Hamburg

Soft Actor-Critic: Deep Reinforcement Learning for Robotics Finn Rietz University of Hamburg

MIN Faculty Department of Informatics Soft Actor-Critic: Deep Reinforcement Learning for Robotics Finn Rietz University of Hamburg Faculty of Mathematics, Informatics and Natural Sciences Department of Informatics Technical Aspects of

742 views • 37 slides
Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning for robots? slide from V. Vanhoucke End-to-end Deep Learning for robots? 2010 : Speech Recognition Audio Acoustic Model Phonetic Model

1.05k views • 65 slides
Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides
CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

DeepLoco by X. B. Peng, G. Berseth & M. van de Panne CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe University // Spring 2018 Neural Face by Taehoon Kim Previously on CMP784 Generative

895 views • 71 slides
Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina Fragkiadaki Course Logistics Course website : all you need to

1.13k views • 88 slides
Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement

Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement

Reinforcement Learning Q-Learning Deep Q-Learning on Atari Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement Learning Q-Learning Deep Q-Learning on Atari Table of Contents Reinforcement Learning

938 views • 63 slides
Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3. Policy Gradient and Gradient Estimators 4. Q-prop: Sample Efficient Policy Gradient and an Off-policy Critic 5. Model Based Planning in Discrete

769 views • 53 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

683 views • 24 slides
10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell

10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell

10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell October 29, 2018 Reading: Barto & Sutton Chapter 15 Tom Mitchell, October 2018 Outline RL in primates RL in humans Error signals and

448 views • 20 slides

More recommend