random expert distillation for imitation learning
play

Random Expert Distillation For Imitation Learning Ruohan - PowerPoint PPT Presentation

Mar 06, 2024 •157 likes •275 views

Random Expert Distillation For Imitation Learning Ruohan Wang, Carlo Ciliberto, Pierluigi Amadori, Yiannis Demiris ICML 2019 Imitation Learning Teacher Student Policy learning from

  1. Random ¡Expert ¡Distillation ¡For ¡ Imitation ¡Learning Ruohan Wang, ¡Carlo ¡Ciliberto, ¡Pierluigi Amadori, ¡Yiannis ¡Demiris ICML ¡2019

  2. Imitation ¡Learning Teacher Student ⁃ Policy ¡learning ¡from ¡a ¡limited set ¡of ¡expert ¡demonstrations ⁃ Intuitive ¡& ¡efficient ¡skills ¡ transfer ⁃ Captures ¡styles ¡& ¡preferences

  3. Inverse ¡Reinforcement ¡Learning - Generative ¡Adversarial ¡ Expert ¡Trajectories Agent ¡Trajectories Imitation ¡Learning ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ (Ho ¡et ¡al., ¡2015) - Optimization ¡challenges Reward ¡Function - Training ¡instability - sample ¡inefficiency RL ¡Algorithm Agent ¡Policy

  4. Random ¡Expert ¡Distillation ¡(RED) - Directly ¡learns ¡a ¡reward ¡ Expert ¡Trajectories function ¡with ¡Random ¡ Network ¡Distillation ¡(RND) ¡ (Burda et ¡al., ¡2018) Reward ¡Function - Considers ¡how ¡“similar” ¡is ¡ the ¡agent ¡to ¡the ¡expert, ¡ instead ¡of ¡how ¡“different” RL ¡Algorithm Agent ¡Policy

  5. Reward ¡Function , ¡ and ¡ 𝑔 . : ¡ℝ 1 → ℝ 3 Over ¡expert ¡trajectories ¡ 𝐸 = {𝑡 % ¡, 𝑏 % } ¡ %*+ = . 𝜄 ∗ = min . ||𝑔 . 𝑡, 𝑏 ¡− 𝑔 ;3< 𝑡, 𝑏 || = Define ¡the ¡reward ¡as = ) . ∗ 𝑡, 𝑏 ¡− 𝑔 𝑠 𝑡, 𝑏 = exp ¡ (−𝜏||𝑔 ;3< 𝑡, 𝑏 || = The ¡reward ¡asymptotically estimates ¡the ¡support of ¡the ¡expert ¡policy

  6. Mujoco Experiments Hopper HalfCheetah Walker2d Reacher Ant GAIL 3614.2 ¡± 7.2 4515.7 ¡± 549.5 4878.0 ¡± 2848.3 -­‑32.4 ¡± 39.8 3186.8 ¡± 903.6 GMMIL 3309.3 ¡± 26.3 3464.2 ¡± 476.5 2967.1 ¡± 702.0 -­‑11.89 ¡± 5.27 991 ± 2.6 RED 3626.0 ¡± 4.3 3072.0 ¡± 84.7 4481.4 ¡± 20.9 -­‑10.43 ¡± 5.2 3552.8 ¡± 348.7 Image ¡ref: ¡https://creativestudio2019spring.files.wordpress.com/2019/02/openaigym.png

  7. Training ¡Stability ¡& ¡Sample ¡Efficiency Hopper Reacher

  8. Driving ¡Task Average Best BC 1033 ¡± 474 1956 GAIL 795 ¡± 395 1576 GMMIL 2024 ¡± 981 3624 RED 4825 ¡± 1552 7485 Expert 7485 ¡± 0 7485

  9. Reward ¡function ¡penalizes ¡dangerous ¡driving

  10. Summary ⁃ Random ¡Expert ¡Distillation ¡is ¡a ¡new ¡framework ¡for ¡imitation ¡learning, ¡ using ¡the ¡estimated ¡support ¡of ¡the ¡expert ¡policy ¡as ¡reward. ⁃ Our ¡results ¡suggest ¡that ¡RED ¡is ¡viable, ¡robust ¡and ¡attains ¡good ¡ performance. ⁃ Future ¡works: ¡combining ¡different ¡sources ¡of ¡expert ¡information ¡for ¡ more ¡robust ¡algorithms.

  11. Thank ¡you ⁃ Code: ¡https://github.com/RuohanW/RED ⁃ Check ¡out ¡our ¡poster: Pacific ¡Ballroom ¡#39 6:30 ¡to ¡9:00 ¡pm ¡today

Recommend

to No-Regret Online Learning Stephane Ross Joint work with Drew Bagnell &amp; Geoff Gordon

to No-Regret Online Learning Stephane Ross Joint work with Drew Bagnell &amp; Geoff Gordon

Reduction of Imitation Learning to No-Regret Online Learning Stephane Ross Joint work with Drew Bagnell &amp; Geoff Gordon Imitation Learning Machine Expert Learning Policy Demonstrations Algorithm 2 Imitation Learning Many

811 views • 46 slides
InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations Chih-Hui Ho, Chun Hu,

InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations Chih-Hui Ho, Chun Hu,

InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations Chih-Hui Ho, Chun Hu, Po-Jung Lai 1 Outline 1. Introduction 2. Related work Generative adversarial imitation learning (GAIL) 3. Proposed method 4. Experiment

692 views • 14 slides
Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &amp;

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &amp;

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &amp; Imitation) Goal: Find Optimal Policy State/Context s t Agent Imitation Learning: Optimize imitation loss Reinforcement Learning: Optimize

547 views • 53 slides
Trajectory Optimization, Imitation Learning Lecture 14 What will you take home today? Recap LQR

Trajectory Optimization, Imitation Learning Lecture 14 What will you take home today? Recap LQR

Trajectory Optimization, Imitation Learning Lecture 14 What will you take home today? Recap LQR Trajectory Optimization Paper Imitation Learning Supervised Learning Dagger How to solve Optimal Control Problems? Sequential Quadratic

750 views • 31 slides
CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised &amp; Imitation

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised &amp; Imitation

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised &amp; Imitation Learning Instructor: Animesh Garg TA: Dylan Turpin &amp; Tingwu Wang Agenda Invitation to Imitation DAGGER: Dataset Aggregation

904 views • 41 slides
Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slide 1

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slide 1

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slide 1 ICML-99 Slides: Dana Dahlstrom CSE 254, UCSD 2002.04.23 Overview Learning by imitation entails watching a mentor perform a task. Slide 2

414 views • 14 slides
One-Shot Imitation Learning Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas

One-Shot Imitation Learning Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas

One-Shot Imitation Learning Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, Wojciech Zaremba Motivation &amp; Problem - Imitation Learning commonly applied to isolated tasks - Desire:

555 views • 21 slides
Formal terms 1 Object-learning Social imitation Emulation Mimicry copying only an

Formal terms 1 Object-learning Social imitation Emulation Mimicry copying only an

24/03/2015 Formal terms 1 Object-learning Social imitation Emulation Mimicry copying only an action form Affordance learning What is imitation? Overimitation copying unnecessary actions 2 Antonia Hamilton UCL 3

542 views • 5 slides
Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slides:

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slides:

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slides: Dana Dahlstrom CSE 254, UCSD 2002.04.23 1 Overview In imitation, a learner observes a mentor in action. The approach proposed in this paper

440 views • 30 slides
Imitation Learning Initial Concept and Approaches Nguyen, Thi Linh Chi Outline Motivation

Imitation Learning Initial Concept and Approaches Nguyen, Thi Linh Chi Outline Motivation

Imitation Learning Initial Concept and Approaches Nguyen, Thi Linh Chi Outline Motivation Basics and Definition Approaches &amp; Examples Conclusion Nguyen, Thi Linh Chi Imitation Learning 2 Motivation Imitation Learning

587 views • 26 slides
A Bayesian Approach to Generative Adversarial Imitation Learning NeurIPS 2018 Presenter Wonseok

A Bayesian Approach to Generative Adversarial Imitation Learning NeurIPS 2018 Presenter Wonseok

A Bayesian Approach to Generative Adversarial Imitation Learning NeurIPS 2018 Presenter Wonseok Jeon @ KAIST Joint work with Seokin Seo @ KAIST Kee-Eung Kim @ KAIST &amp; PROWLER.io Imitation Learning A Markov decision process (MDP)

431 views • 14 slides
Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent state reward action S t R t A t R t+ 1 Environment S t+ 1

1.07k views • 52 slides
10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018

10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018

10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018 Recommended readings: Used Materials Much of the material and slides for this lecture were borrowed from Katerina Fragkiadaki, and Ruslan

1.25k views • 49 slides
Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Lecture 7: Imitation Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning 3 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 /

1.27k views • 45 slides
Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Lecture 7: Imitation Learning in Large State Spaces 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2020 1 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning in Large State Spaces 1 Emma Brunskill (CS234

797 views • 57 slides
Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Lecture 7: Imitation Learning in Large State Spaces 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2020 1 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning in Large State Spaces 1 Emma Brunskill (CS234

526 views • 52 slides
CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G.,

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G.,

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G., &amp; Stone, P. (2018). Behavioral cloning from observation. In IJCAI (pp. 4950-4957). Ho, J., &amp; Ermon, S. (2016). Generative adversarial

464 views • 14 slides
Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement

Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement Learning Probability Dynamics distribution over next Model T

911 views • 66 slides
Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement

Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement Learning Probability Dynamics distribution over next Model T

608 views • 56 slides
Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Lecture 7: Imitation Learning in Large State Spaces 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2019 2 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning in Large State Spaces 3 Emma Brunskill (CS234

924 views • 47 slides
Gen enerativ erative e Adver ersaria sarial l Im Imitation itation Le Learning arning

Gen enerativ erative e Adver ersaria sarial l Im Imitation itation Le Learning arning

Gen enerativ erative e Adver ersaria sarial l Im Imitation itation Le Learning arning Stefan efano o Ermon on Joint work with Jayesh Gupta, Jonathan Ho, Yunzhu Li, Hongyu Ren, and Jiaming Song Reinforcement Learning Goal: Learn

530 views • 40 slides
Imitation Learning from Imperfect Demonstration Yueh-Hua Wu 1,2 , Nontawat Charoenphakdee 3,2 , Han

Imitation Learning from Imperfect Demonstration Yueh-Hua Wu 1,2 , Nontawat Charoenphakdee 3,2 , Han

Imitation Learning from Imperfect Demonstration Yueh-Hua Wu 1,2 , Nontawat Charoenphakdee 3,2 , Han Bao 3,2 , Voot Tangkaratt 2 , Masashi Sugiyama 2,3 1 National Taiwan University 2 RIKEN Center for Advanced Intelligence Project 3 The University of

309 views • 12 slides
One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning Authors: Tianhe Yu*,

One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning Authors: Tianhe Yu*,

One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning Authors: Tianhe Yu*, Chelsea Finn*, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, Sergey Levine CS 330: Deep Multi-Task and Meta-Learning October 16, 2019

432 views • 24 slides
Imitation Theory and Experimental Evidence Joerg Oechssler University of Heidelberg

Imitation Theory and Experimental Evidence Joerg Oechssler University of Heidelberg

Imitation Theory and Experimental Evidence Joerg Oechssler University of Heidelberg Imitation is relevant Neoplan (Germany) vs. Zhongwei (P.R. China) Imitation is prevalent Zeiss Ikon Contax II (1936) vs. Nikon I (1948) Imitator

655 views • 46 slides

More recommend