modular mul task reinforcement learning with policy
play

Modular mul*task reinforcement learning with policy sketches Jacob - PowerPoint PPT Presentation

Oct 03, 2022 •634 likes •1.18k views

Modular mul*task reinforcement learning with policy sketches Jacob Andreas, Sergey Levine and Dan Klein The learning problem make planks 2 The learning problem make planks make sticks 3 Learning from sketches get wood get wood use saw

  1. Modular mul*task reinforcement learning with policy sketches Jacob Andreas, Sergey Levine and Dan Klein

  2. The learning problem make planks 2

  3. The learning problem make planks make sticks 3

  4. Learning from sketches get wood get wood use saw use axe 4

  5. The op*ons framework 5

  6. The op*ons framework +1 6

  7. The op*ons framework +1 7

  8. The op*ons framework [SuCon et al. 99, Bacon & Precup 16] 8

  9. Learning from intermediate rewards r r [Kearns & Singh 02, Kulkarni et al. 16] 9

  10. Learning from demonstra*ons Ï [Stolle & Precup 02, Fox & Krishnan et al. 16] 10

  11. Learning from policy sketches get wood use saw Ï 11

  12. Why sketches? Easy to collect Portable Crafting environment make plank get wood use toolshed make stick get wood use workbench make cloth get grass use factory make rope get grass use toolshed make bridge get iron get wood make bed ∗ get wood use toolshed make axe ∗ get wood use workbench make shears get wood use workbench get gold get iron get wood get gem get wood use workbench 12

  13. Learning from policy sketches

  14. Learning from policy sketches make planks get wood use saw 14

  15. Learning from policy sketches make sticks get wood use axe 15

  16. Learning from policy sketches get wood π a use saw get wood π b use axe 16 [e.g. Branavan et al. 09, Oh et al. 17, Hermann et al. 17]

  17. Learning from policy sketches get wood use saw get wood use axe 17

  18. ` get wood use saw π 1 π 2 get wood use axe π 1 π 3 18

  19. ` get wood use saw π 1 π 2 get wood use axe π 1 π 3 19

  20. ` get wood π 1

  21. Policy representa*on π 1 get wood 21

  22. Policy representa*on ??? π 1 get wood 22

  23. Policy representa*on 23

  24. Policy representa*on 24

  25. Policy representa*on 25

  26. Policy representa*on Ac*on probabili*es π 1 get wood 26

  27. Policy search ac*on state reward baseline Σ Σ ( ) ∇ log π( | ) (r t - b) tasks steps 27

  28. Policy search Σ Σ ( ) ∇ log π( | ) (r t - b) tasks steps get wood 28

  29. Policy search Σ Σ ( ) ∇ log π( | ) (r t - b) tasks steps use axe 29

  30. Policy search Reward .40 Σ Σ ( ) ∇ log π( | ) (r t - b) SUBPOLICY tasks steps 30

  31. Improving policy search 31

  32. Improving policy search ac*on state reward baseline Σ Σ ( ) ∇ log π( | ) (r t - b) tasks steps 32

  33. Improving policy search ( ) ( ) ∇ log π( | ) (r t - ) ∇ log π( | ) (r t - ) use saw use saw make planks make nails ( ) ( ) ∇ log π( | ) (r t - ) ∇ log π( | ) (r t - ) use axe use axe make planks make nails ( ) ( ) ∇ log π( | ) (r t - ) ∇ log π( | ) (r t - ) get wood get wood make planks make nails ( ) ( ) ∇ log π( | ) (r t - ) ∇ log π( | ) (r t - ) get iron get iron make planks make nails 33

  34. Improving policy search .89 Reward .40 Σ Σ ( ) ∇ log π( | ) (r t - ) SUBPOLICY TASK tasks steps 34

  35. Do sketches help?

  36. The maze naviga*on task 36

  37. The maze naviga*on task 37

  38. The maze naviga*on task Sketches: modular Unsupervised Reward Sketches: joint 0 1 2 3 x 10 6 episodes 38

  39. The mini-crag task 39

  40. The mini-crag task 40

  41. The mini-crag task Sketches: modular Sketches: joint Reward Unsupervised 0 1 2 3 x 10 6 episodes 41

  42. The cliff-walking task 42

  43. The cliff-walking task Sketches: modular log Reward Sketches: joint Unsupervised 0 1 2 3 x 10 8 *mesteps 43

  44. Zero-shot generaliza*on What if I see a sketch I’ve never seen before? get iron use axe 44

  45. Zero-shot generaliza*on What if I see a sketch I’ve never seen before? 100 89 75 Joint 77 50 Modular 49 25 1 0 Mul*task Zero-shot 45

  46. Zero-shot generaliza*on What if I see a sketch I’ve never seen before? 100 89 75 Joint 77 50 Modular 49 25 1 0 Mul*task Zero-shot 46

  47. Fast adapta*on What if I don’t get a sketch at test *me? ??? 47

  48. Fast adapta*on What if I don’t get a sketch at test *me? 100 89 75 Unsupervised 77 50 Sketches 47 25 1 0 Mul*task Adapta*on 48

  49. Fast adapta*on What if I don’t get a sketch at test *me? 100 89 75 Unsupervised 76 50 Sketches 47 42 25 0 Mul*task Adapta*on 49

  50. Conclusions

  51. A *ny bit of data goes a long way Crafting environment make plank get wood use toolshed make stick get wood use workbench make cloth get grass use factory make rope get grass use toolshed make bridge get iron get wood use factory make bed ∗ get wood use toolshed get grass use workbench make axe ∗ get wood use workbench get iron use toolshed make shears get wood use workbench get iron use workbench get gold get iron get wood use factory use bridge get gem get wood use workbench get iron use toolshed use axe 51

  52. A *ny bit of data goes a long way Crafting environment make plank get wood use toolshed make stick get wood use workbench make cloth get grass use factory make rope get grass use toolshed make bridge get iron get wood use factory make bed ∗ get wood use toolshed get grass use workbench make axe ∗ get wood use workbench get iron use toolshed make shears get wood use workbench get iron use workbench get gold get iron get wood use factory use bridge get gem get wood use workbench get iron use toolshed use axe 52

  53. Thank you! https://github.com/jacobandreas/psketch

Recommend

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3. Policy Gradient and Gradient Estimators 4. Q-prop: Sample Efficient Policy Gradient and an Off-policy Critic 5. Model Based Planning in Discrete

769 views • 53 slides
Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement & Imitation) Goal: Find Optimal Policy State/Context s t Agent Imitation Learning: Optimize imitation loss Reinforcement Learning: Optimize

547 views • 53 slides
Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

719 views • 40 slides
Structured Losses Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

Structured Losses Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

Structured Losses Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning Junhyuk Oh, Satinder Singh, Honglak Lee, Pushmeet Kohli Oh et al. 2017 https://arxiv.org/abs/1706.05064 Presented by Beln Saldas belen@mit.edu

820 views • 51 slides
Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides
Meta Reinforcement Learning as Task Inference Jan Humplik, Alexandre Galashov, Leonard

Meta Reinforcement Learning as Task Inference Jan Humplik, Alexandre Galashov, Leonard

Meta Reinforcement Learning as Task Inference Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A.Ortega, Yee Whye Teh, Nicholas Heess Topic: Bayesian RL Presenter: Ram Ananth Why meta Reinforcement Learning? First Wave of Deep

912 views • 63 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

683 views • 24 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

541 views • 15 slides
Transfer and Multi-Task Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class

Transfer and Multi-Task Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class

Transfer and Multi-Task Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. The project milestone is next week! 2. HW4 due tonight! 3. HW5 releases shortly (Wed or Fri) Three different options: maximum entropy

773 views • 50 slides
Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost

Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost

Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost using data Solid math for the tuning DNNs support approximations Safety? Explainability? 8 th of November, 2019 S. Gros SARLEM 1 / 1 Reinforcement

311 views • 9 slides
Reinforcement Learning Policy Op5miza5on Pieter Abbeel UC Berkeley EECS Policy

Reinforcement Learning Policy Op5miza5on Pieter Abbeel UC Berkeley EECS Policy

Reinforcement Learning Policy Op5miza5on Pieter Abbeel UC Berkeley EECS Policy Op>miza>on n Consider control policy parameterized by parameter vector H X max E[ R ( s t ) | ] t =0 n O<en stochas>c policy class

762 views • 36 slides
Public Policy and Deep Reinforcement Learning on AWS Emily Webber | Machine Learning Specialist

Public Policy and Deep Reinforcement Learning on AWS Emily Webber | Machine Learning Specialist

Public Policy and Deep Reinforcement Learning on AWS Emily Webber | Machine Learning Specialist at Amazon Web Services | To-be-open-sourced research project Public Policy Has Unique Challenges Structural Inefficiency Lack of single goal

387 views • 20 slides
1 Video of Demo Q-Learning Auto Cliff Grid Exploration vs. Exploitation How to Explore? Video

1 Video of Demo Q-Learning Auto Cliff Grid Exploration vs. Exploitation How to Explore? Video

Reinforcement Learning Reinforcement Learning II We still assume an MDP: A set of states s S A set of actions (per state) A A model T(s,a,s) A reward function R(s,a,s) Still looking for a policy (s) New

358 views • 6 slides
Reinforcement Learning: A Primer, Multi-Task, Goal-Conditioned CS 330 1 Logistics Homework

Reinforcement Learning: A Primer, Multi-Task, Goal-Conditioned CS 330 1 Logistics Homework

Reinforcement Learning: A Primer, Multi-Task, Goal-Conditioned CS 330 1 Logistics Homework 2 due Wednesday. Homework 3 out on Wednesday. Project proposal due next Wedesday. 2 Why Reinforcement Learning? When do you not need sequential

895 views • 44 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
Autonomous(Task(Sequencing(for(Customized( Curriculum(Design(in(Reinforcement(Learning

Autonomous(Task(Sequencing(for(Customized( Curriculum(Design(in(Reinforcement(Learning

Autonomous(Task(Sequencing(for(Customized( Curriculum(Design(in(Reinforcement(Learning Sanmit'Narvekar, Jivko Sinapov,+and+Peter+Stone Department+of+Computer+Science University+of+Texas+at+Austin {sanmit,+jsinapov,+pstone}+@cs.utexas.edu

620 views • 26 slides
Task-Agnostic Dynamics Priors for Deep Reinforcement Learning Yilun Du 1 , Karthik Narasimhan 2

Task-Agnostic Dynamics Priors for Deep Reinforcement Learning Yilun Du 1 , Karthik Narasimhan 2

Task-Agnostic Dynamics Priors for Deep Reinforcement Learning Yilun Du 1 , Karthik Narasimhan 2 1 MIT, 2 Princeton Key Questions t t+1 Can we learn physics in a task-agnostic fashion? Does it help sample efficiency of RL? Can we

555 views • 14 slides
Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring environment. Interactions with environment: action state Agent Environment reward Problem: find action policy that maximizes cumulative reward

828 views • 66 slides
Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly, K., Zhou, A., Quillen, D., Finn, C., & Levine, S ICML, 2019 Presented by: Egill Ian Gudmundsson Some Terminology On-policy learning: Only one

427 views • 16 slides
Reinforcement Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 13

Reinforcement Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 13

0. Reinforcement Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 13 Acknowledgement: The present slides are an adaptation of slides drawn by T. Mitchell 1. Reinforcement Learning Overview Task: Control

396 views • 24 slides
Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an

Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an

1 Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an action Lecture 2 We get a reward Gillian Hayes Our aim is to learn a policy what action to choose in what state to get maximum reward 11th

641 views • 5 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides

More recommend