Trajectory Optimization, Imitation Learning Lecture 14 What will - PowerPoint PPT Presentation

Trajectory Optimization, Imitation Learning Lecture 14

What will you take home today? Recap LQR Trajectory Optimization Paper Imitation Learning Supervised Learning Dagger

How to solve Optimal Control Problems?

Sequential Quadratic Programming

Example – Newton-Raphson Method

Sequential Linear Quadratic Programming

SLQ Algorithm

Linear Dynamical Systems, Quadratic cost – L inear Q uadratic R egulator (LQR)

Trajectory Optimization

What will you take home today? Recap LQR Trajectory Optimization Paper Imitation Learning Supervised Learning Dagger

Assumptions in Optimal Control 1. Known and/or simple System Dynamics 2. Known Cost function

What are approaches for unknown dynamics and/or cost? 1. Learning approaches a. Reinforcement learning Model-based i. Model-based ii. b. Imitation learning Imitate an expert policy i.

Learning to make single predictions versus a sequence of predictions

Running Example: Super Tux Cart from A Reduction of Imitation Learning and Structured Predictionto No-Regret Online Learning. Ross, Gordon, Bagnell. AIStats. 2011. https://www.youtube.com/watch?feature=oembed&v=V00npNnWzSU

Imitation Learning 1. Useful when dynamics and/or cost are unknown/complex a. We don’t know how the next state will look like. Hard to model. b. We don’t know the cost to go for an action The expected immediate cost of taking action a in state s The expected immediate cost of executing policy pi in state s Cost-to-go = total cost of executing pi over T

Imitation learning – core idea 1. Idea: imitate expert trajectories! a. Bound J for any cost function C based on how well pi mimics expert’s policy

Imitation Learning by Classification Algorithm from - A Course in Machine Learning by Hal Daumé III. Ch. 18

How well does Imitation Learning by Classification work? 1. Depends on a. How good the expert is. b. How much error the classifier makes.

Running Example: Super Tux Cart Figure from ‘Interactive Learning for Sequential Decisions and Predictions’ by Stephane Ross.

Learned behavior influences states and observations Challenge: system dynamics are assumed both unknown and complex, we cannot ● compute dπ and can only sample it by executing π in the system. non-i.i.d. supervised learning problem due to the dependence of the input distribution on ● the policy π itself. Difficult optimization due to dependence which makes problem non-convex Typical assumption in statistics and machine learning: Observations in a sample are independent and identically distributed. This simplifies many methods although not true in many practical settings. Examples are coin flips. Roulette spins When are data samples i.i.d?

Running Example: Super Tux Cart from A Reduction of Imitation Learning and Structured Predictionto No-Regret Online Learning. Ross, Gordon, Bagnell. AIStats. 2011. https://www.youtube.com/watch?feature=oembed&v=V00npNnWzSU

Another example – Super Mario from A Reduction of Imitation Learning and Structured Predictionto No-Regret Online Learning. Ross, Gordon, Bagnell. AIStats. 2011. https://www.youtube.com/watch?v=anOI0xZ3kGM

How do we train a policy that can deal with any possible situation? 1. This is impossible since the state/observation space may be prohibitively large and we cannot train on allpossible configurations. If we could, we may just memorize anyway. 2. Goal: Train f to do well on configurations that it encounters itself. 3. Chicken and egg problem: a. We want a policy that does well in a bunch of world configuration. b. What configurations? The ones it encounters/ 4. Solve by iteration: roll out f. Collect data, retrain.

Dataset Aggregation Algorithm (Dagger) Figure and Algorithm from - A Course in Machine Learning by Hal Daumé III. Ch. 18

How well does Dagger work? Theorem from - A Course in Machine Learning by Hal Daumé III. Ch. 18

Running Example: Super Tux Cart from A Reduction of Imitation Learning and Structured Predictionto No-Regret Online Learning. Ross, Gordon, Bagnell. AIStats. 2011.

Requirements on the Expert 1. Human demonstrations 2. Expensive but exact algorithm that is too slow to run in real time

Trajectory Optimization, Imitation Learning Lecture 14 What will - PowerPoint PPT Presentation

Trajectory Optimization, Imitation Learning Lecture 14 What will you take home today? Recap LQR Trajectory Optimization Paper Imitation Learning Supervised Learning Dagger How to solve Optimal Control Problems? Sequential Quadratic

Optimal Control, LQR, Trajectory Optimization Lecture 13 What will you take home today? Intro

Optimization Based Iterative Learning Control for Trajectory Tracking Angela Schoellig and

Random Expert Distillation For Imitation Learning Ruohan Wang, Carlo

InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations Chih-Hui Ho, Chun Hu,

Adaptive Trajectory Optimization Gregory Kahn et al., ICRA 2017 SeungWoon Kim Probabilistic 3D

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised & Imitation

Optimization-Based Control: Direct Collocation Methods for Trajectory and Policy Optimization CS

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slide 1

One-Shot Imitation Learning Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas

Formal terms 1 Object-learning Social imitation Emulation Mimicry copying only an

SCP for trajectory optimization Basic problem minimize_{traj} path_length + other costs

to No-Regret Online Learning Stephane Ross Joint work with Drew Bagnell & Geoff Gordon

From trajectory optimization to inverse KKT and sequential manipulation Marc Toussaint Machine

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slides:

Imitation Learning Initial Concept and Approaches Nguyen, Thi Linh Chi Outline Motivation

Trajectory Option Set Planning Optimization in CTOP Leonardo Cruciol TransLab, UnB Outline

A Bayesian Approach to Generative Adversarial Imitation Learning NeurIPS 2018 Presenter Wonseok

Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Trajectory Optimization (this is a draft, to be updated before lecture) McGill COMP 765 Oct 5 th

10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Trajectory Optimization, Imitation Learning Lecture 14 What will - PowerPoint PPT Presentation

Trajectory Optimization, Imitation Learning Lecture 14 What will you take home today? Recap LQR Trajectory Optimization Paper Imitation Learning Supervised Learning Dagger How to solve Optimal Control Problems? Sequential Quadratic

Optimal Control, LQR, Trajectory Optimization Lecture 13 What will you take home today? Intro

Optimization Based Iterative Learning Control for Trajectory Tracking Angela Schoellig and

Random Expert Distillation For Imitation Learning Ruohan Wang, Carlo

InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations Chih-Hui Ho, Chun Hu,

Adaptive Trajectory Optimization Gregory Kahn et al., ICRA 2017 SeungWoon Kim Probabilistic 3D

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &amp;

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised &amp; Imitation

Optimization-Based Control: Direct Collocation Methods for Trajectory and Policy Optimization CS

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slide 1

One-Shot Imitation Learning Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas

Formal terms 1 Object-learning Social imitation Emulation Mimicry copying only an

SCP for trajectory optimization Basic problem minimize_{traj} path_length + other costs

to No-Regret Online Learning Stephane Ross Joint work with Drew Bagnell &amp; Geoff Gordon

From trajectory optimization to inverse KKT and sequential manipulation Marc Toussaint Machine

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slides:

Imitation Learning Initial Concept and Approaches Nguyen, Thi Linh Chi Outline Motivation

Trajectory Option Set Planning Optimization in CTOP Leonardo Cruciol TransLab, UnB Outline

A Bayesian Approach to Generative Adversarial Imitation Learning NeurIPS 2018 Presenter Wonseok

Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Trajectory Optimization (this is a draft, to be updated before lecture) McGill COMP 765 Oct 5 th

10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised & Imitation

to No-Regret Online Learning Stephane Ross Joint work with Drew Bagnell & Geoff Gordon