improving imitation learning with reinforcement learning
play

Improving Imitation Learning with Reinforcement Learning Niklas - PowerPoint PPT Presentation

Mar 25, 2024 •570 likes •905 views

MIN Faculty Department of Informatics Improving Imitation Learning with Reinforcement Learning Niklas Fiedler University of Hamburg Faculty of Mathematics, Informatics and Natural Sciences Department of Informatics Technical Aspects of

  1. MIN Faculty Department of Informatics Improving Imitation Learning with Reinforcement Learning Niklas Fiedler University of Hamburg Faculty of Mathematics, Informatics and Natural Sciences Department of Informatics Technical Aspects of Multimodal Systems November 26, 2019 N. Fiedler – Improving Imitation Learning with Reinforcement Learning 1 / 23

  2. Outline Introduction Imitation Learning Combining RL and IL Conclusion 1. Introduction Motivation 2. Imitation Learning Demonstration Methods Behavioral Cloning Inverse Reinforcement Learning 3. Combining Reinforcement Learning and Imitation Learning BC Application IRL Application 4. Conclusion N. Fiedler – Improving Imitation Learning with Reinforcement Learning 2 / 23

  3. Goal Introduction Imitation Learning Combining RL and IL Conclusion ◮ Imitate expert behavior ◮ Improve learning by including knowledge given by demonstration ◮ Learn expert policies → Make use of expert demonstrations N. Fiedler – Improving Imitation Learning with Reinforcement Learning 3 / 23

  4. Motivation Humans are Awesome Introduction Imitation Learning Combining RL and IL Conclusion https://rejectedprincesses.tumblr.com/post/150495232038/ chynara-madinkulova-long-hair-and-aida-akmatova N. Fiedler – Improving Imitation Learning with Reinforcement Learning 4 / 23

  5. Motivation Learning from Demonstration Introduction Imitation Learning Combining RL and IL Conclusion Learning from experts is natural behavior [Haw50], https://www.wakecounseling.com/therapy-blog/play-therapy N. Fiedler – Improving Imitation Learning with Reinforcement Learning 5 / 23

  6. Imitation Learning Introduction Imitation Learning Combining RL and IL Conclusion Method to learn a behavior based on a demonstration Various forms of demonstration. Two prominent methods of implementation: 1. Behavioral Cloning 2. Inverse Reinforcement Learning N. Fiedler – Improving Imitation Learning with Reinforcement Learning 6 / 23

  7. Imitation Learning Introduction Imitation Learning Combining RL and IL Conclusion Method to learn a behavior based on a demonstration Various forms of demonstration. Two prominent methods of implementation: 1. Behavioral Cloning 2. Inverse Reinforcement Learning N. Fiedler – Improving Imitation Learning with Reinforcement Learning 6 / 23

  8. Imitation Learning Introduction Imitation Learning Combining RL and IL Conclusion Method to learn a behavior based on a demonstration Various forms of demonstration. Two prominent methods of implementation: 1. Behavioral Cloning 2. Inverse Reinforcement Learning N. Fiedler – Improving Imitation Learning with Reinforcement Learning 6 / 23

  9. Demonstration Methods Introduction Imitation Learning Combining RL and IL Conclusion Virtual/Augumented Reality Tracking of Human Motions Video Stream Teleoperation s3.ap-south-1.amazonaws.com/kidobotikz.sprw/master/assets/images/blog/blog-2018110811630.jpg siamagazin.com/bimanual-teleoperation-of-a-compliant-whole-body-controlled-humanoid-robot/ https://ar-tracking.com/applications/motion-capture/ https://www.youtube.com/watch?v=5BTIE_fhReo N. Fiedler – Improving Imitation Learning with Reinforcement Learning 7 / 23

  10. Demonstration Methods Introduction Imitation Learning Combining RL and IL Conclusion Virtual/Augumented Reality Tracking of Human Motions Video Stream Teleoperation s3.ap-south-1.amazonaws.com/kidobotikz.sprw/master/assets/images/blog/blog-2018110811630.jpg siamagazin.com/bimanual-teleoperation-of-a-compliant-whole-body-controlled-humanoid-robot/ https://ar-tracking.com/applications/motion-capture/ https://www.youtube.com/watch?v=5BTIE_fhReo N. Fiedler – Improving Imitation Learning with Reinforcement Learning 7 / 23

  11. Demonstration Methods Introduction Imitation Learning Combining RL and IL Conclusion Virtual/Augumented Reality Tracking of Human Motions Video Stream Teleoperation s3.ap-south-1.amazonaws.com/kidobotikz.sprw/master/assets/images/blog/blog-2018110811630.jpg siamagazin.com/bimanual-teleoperation-of-a-compliant-whole-body-controlled-humanoid-robot/ https://ar-tracking.com/applications/motion-capture/ https://www.youtube.com/watch?v=5BTIE_fhReo N. Fiedler – Improving Imitation Learning with Reinforcement Learning 7 / 23

  12. Demonstration Methods Introduction Imitation Learning Combining RL and IL Conclusion Virtual/Augumented Reality Tracking of Human Motions Video Stream Teleoperation s3.ap-south-1.amazonaws.com/kidobotikz.sprw/master/assets/images/blog/blog-2018110811630.jpg siamagazin.com/bimanual-teleoperation-of-a-compliant-whole-body-controlled-humanoid-robot/ https://ar-tracking.com/applications/motion-capture/ https://www.youtube.com/watch?v=5BTIE_fhReo N. Fiedler – Improving Imitation Learning with Reinforcement Learning 7 / 23

  13. Behavioral Cloning Introduction Imitation Learning Combining RL and IL Conclusion ◮ Training a direct link between demonstrated input and output ◮ Large amounts of training data necessary ◮ Poor generalization N. Fiedler – Improving Imitation Learning with Reinforcement Learning 8 / 23

  14. Behavioral Cloning Video Introduction Imitation Learning Combining RL and IL Conclusion https://www.youtube.com/watch?v=5BTIE_fhReo N. Fiedler – Improving Imitation Learning with Reinforcement Learning 9 / 23

  15. Inverse Reinforcement Learning Reinforcement Learning Introduction Imitation Learning Combining RL and IL Conclusion N. Fiedler – Improving Imitation Learning with Reinforcement Learning 10 / 23

  16. Inverse Reinforcement Learning Reinforcement Learning vs. Inversed Reinforcement Learning Introduction Imitation Learning Combining RL and IL Conclusion RL IRL (partially observed) policy π or history given reward function R sampled from that policy reward function R optimal policy π searching for which given behavior for given reward is optimal https://thinkingwires.com/posts/2018-02-13-irl-tutorial-1.html N. Fiedler – Improving Imitation Learning with Reinforcement Learning 11 / 23

  17. Inverse Reinforcement Learning Introduction Imitation Learning Combining RL and IL Conclusion https://medium.com/@sanketgujar95/generative-adversarial-imitation-learning-266f45634e60 N. Fiedler – Improving Imitation Learning with Reinforcement Learning 12 / 23

  18. Imitation Learning Behavioral Cloning vs. Inversed Reinforcement Learning Introduction Imitation Learning Combining RL and IL Conclusion Inversed Reinforcement Behavioral Cloning Learning ◮ Weak generalization ◮ Strong generalization ◮ Relatively low ◮ Large computational effort computational effort ◮ Complex structure N. Fiedler – Improving Imitation Learning with Reinforcement Learning 13 / 23

  19. Combining Reinforcement Learning and Imitation Learning Introduction Imitation Learning Combining RL and IL Conclusion ◮ Reducing the impact of shortcomings of both methods ◮ Applications should outperform demonstrators after RL applications ◮ Accelerated training process ◮ Extending the capabilities learned with imitation learning N. Fiedler – Improving Imitation Learning with Reinforcement Learning 14 / 23

  20. BC Application Introduction Imitation Learning Combining RL and IL Conclusion Overcoming Exploration in Reinforcement Learning with Demonstrations Ashvin Nair 12 , Bob McGrew 1 , Marcin Andrychowicz 1 , Wojciech Zaremba 1 and Pieter Abbeel 12 2018 IEEE International Conference on Robotics and Automation (ICRA) 1 OpenAI 2 University of California, Berkeley N. Fiedler – Improving Imitation Learning with Reinforcement Learning 15 / 23

  21. BC Application Goal Introduction Imitation Learning Combining RL and IL Conclusion Pushing Sliding Pick and Place [NMA + 18] N. Fiedler – Improving Imitation Learning with Reinforcement Learning 16 / 23

  22. BC Application Results Introduction Imitation Learning Combining RL and IL Conclusion [NMA + 18] N. Fiedler – Improving Imitation Learning with Reinforcement Learning 17 / 23

  23. IRL Application Introduction Imitation Learning Combining RL and IL Conclusion Reinforcement and Imitation Learning for Diverse Visuomotor Skills Yuke Zhu 1 , Ziyu Wang 2 , Josh Merel 2 , Andrei Rusu 2 , Tom Erez 2 , Serkan Cabi 2 , Saran Tunyasuvunakool 2 , Janos Kramar 2 , Raia Hadsell 2 , Nando de Freitas 2 and Nicolas Heess 2 1 Computer Science Department, Stanford University 2 OpenAI N. Fiedler – Improving Imitation Learning with Reinforcement Learning 18 / 23

  24. IRL Application Goal Introduction Imitation Learning Combining RL and IL Conclusion [ZWM + 18] N. Fiedler – Improving Imitation Learning with Reinforcement Learning 19 / 23

  25. IRL Application Method Introduction Imitation Learning Combining RL and IL Conclusion [ZWM + 18] N. Fiedler – Improving Imitation Learning with Reinforcement Learning 20 / 23

  26. IRL Example Results - Block stacking Introduction Imitation Learning Combining RL and IL Conclusion [ZWM + 18] N. Fiedler – Improving Imitation Learning with Reinforcement Learning 21 / 23

  27. Combining Reinforcement Learning and Imitation Learning Comparison Introduction Imitation Learning Combining RL and IL Conclusion IRL Approach BC Approach ◮ Inversed Reinforcement ◮ Behavioral Cloning Learning ◮ Simulation only ◮ Policies transferred to real robot ◮ Goal : improve training ◮ Goal : improve result performance and task complexity performance and task complexity N. Fiedler – Improving Imitation Learning with Reinforcement Learning 22 / 23

Recommend

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement & Imitation) Goal: Find Optimal Policy State/Context s t Agent Imitation Learning: Optimize imitation loss Reinforcement Learning: Optimize

547 views • 53 slides
Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent state reward action S t R t A t R t+ 1 Environment S t+ 1

1.07k views • 52 slides
Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Lecture 7: Imitation Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning 3 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 /

1.27k views • 45 slides
Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement

Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement Learning Probability Dynamics distribution over next Model T

608 views • 56 slides
Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement

Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement Learning Probability Dynamics distribution over next Model T

911 views • 66 slides
CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised & Imitation

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised & Imitation

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised & Imitation Learning Instructor: Animesh Garg TA: Dylan Turpin & Tingwu Wang Agenda Invitation to Imitation DAGGER: Dataset Aggregation

904 views • 41 slides
Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slide 1

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slide 1

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slide 1 ICML-99 Slides: Dana Dahlstrom CSE 254, UCSD 2002.04.23 Overview Learning by imitation entails watching a mentor perform a task. Slide 2

414 views • 14 slides
10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018

10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018

10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018 Recommended readings: Used Materials Much of the material and slides for this lecture were borrowed from Katerina Fragkiadaki, and Ruslan

1.25k views • 49 slides
CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G.,

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G.,

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G., & Stone, P. (2018). Behavioral cloning from observation. In IJCAI (pp. 4950-4957). Ho, J., & Ermon, S. (2016). Generative adversarial

464 views • 14 slides
Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slides:

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slides:

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slides: Dana Dahlstrom CSE 254, UCSD 2002.04.23 1 Overview In imitation, a learner observes a mentor in action. The approach proposed in this paper

440 views • 30 slides
Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Lecture 7: Imitation Learning in Large State Spaces 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2019 2 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning in Large State Spaces 3 Emma Brunskill (CS234

924 views • 47 slides
Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Lecture 7: Imitation Learning in Large State Spaces 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2020 1 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning in Large State Spaces 1 Emma Brunskill (CS234

526 views • 52 slides
Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Lecture 7: Imitation Learning in Large State Spaces 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2020 1 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning in Large State Spaces 1 Emma Brunskill (CS234

797 views • 57 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Gen enerativ erative e Adver ersaria sarial l Im Imitation itation Le Learning arning

Gen enerativ erative e Adver ersaria sarial l Im Imitation itation Le Learning arning

Gen enerativ erative e Adver ersaria sarial l Im Imitation itation Le Learning arning Stefan efano o Ermon on Joint work with Jayesh Gupta, Jonathan Ho, Yunzhu Li, Hongyu Ren, and Jiaming Song Reinforcement Learning Goal: Learn

530 views • 40 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides
Random Expert Distillation For Imitation Learning Ruohan Wang, Carlo

Random Expert Distillation For Imitation Learning Ruohan Wang, Carlo

Random Expert Distillation For Imitation Learning Ruohan Wang, Carlo Ciliberto, Pierluigi Amadori, Yiannis Demiris ICML 2019 Imitation Learning Teacher Student Policy learning from

275 views • 11 slides
Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement

Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement

Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement Learning Quentin Cappart , Emmanuel Goutierre, David Bergman, Louis-Martin Rousseau 1 Research question Bounding mechanisms are critical in the

449 views • 24 slides
Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Machine Learning Supervised Learning Teacher tells learner what to remember Reinforcement Learning

401 views • 29 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

682 views • 24 slides
Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides

More recommend