efficient reinforcement learning with hierarchies of
play

Efficient Reinforcement Learning with Hierarchies of Machines by - PowerPoint PPT Presentation

Aug 23, 2023 •98 likes •273 views

Efficient Reinforcement Learning with Hierarchies of Machines by Leveraging Internal Transitions Aijun Bai* Stuart Russell UC Berkeley/Microsoft Research UC Berkeley

  1. Efficient Reinforcement Learning with Hierarchies of Machines by Leveraging Internal Transitions Aijun Bai* Stuart Russell UC Berkeley/Microsoft Research UC Berkeley

  2. Outline • Hierarchical RL with partial programs • Deterministic internal transitions • Results

  3. Hierarchical RL with partial programs [Parr & Russell, NIPS 97; Andre & Russell, NIPS 00, AAAI 02; Marthi et al, IJCAI 05] Partial program a Learning algorithm s,r Completion Hierarchically optimal for all terminating programs 3

  4. Partial Program – an Example repeat forever Choose({a1,a2,…})

  5. Partial Program – an Example Navigate(destination) while  At(destination,CurrentState()) Choose({N,S,E,W})

  6. Concurrent Partial Programs Top() for each p in Effectors() PlayKeep(p) PlayKeep(p) s  CurrentState() while  Terminal(s) if BallKickable(s) then Choose({Pass(),Hold()}) else if FastestToBall(s) then Intercept() else Choose(Stay(),Move()) Pass() KickTo(Choose(Effectors()\{self}),Choose({slow,fast}) …

  7. Technical development • Decisions based on internal state • Joint state ω = [s,m] environment state + program state (cf. [Russell & Wefald 1989] ) • MDP + partial program = SMDP over choice states in {ω}, learn Q(ω,c) for choices c • Additive decomposition of value functions • by subroutine structure [Dietterich 00, Andre & Russell 02] Q is a sum of sub-Q functions per subroutine • across concurrent threads [Russell & Zimdars 03] Q is a sum of sub-Q functions per thread, with decomposed reward signal 7

  8. Internal Transitions Top() • Transitions between for each p in Effectors() choice points with no PlayKeep(p) physical action intervening PlayKeep(p) s  CurrentState() • Internal transitions while  Terminal(s) take no (real) time and have zero reward if BallKickable(s) then Choose({Pass(),Hold()}) else if FastestToBall(s) then Intercept() • Internal transitions are else Choose(Stay(),Move()) deterministic Pass() KickTo(Choose(Effectors()\{self}),Choose({slow,fast}) …

  9. Idea 1 • Use internal transitions to shortcircuit the computations of Q values recursively if applicable • If (s, m, c) -> (s, m’) is an internal transition • Then, Q(s, m, c) = V(s, m’) = max c’ Q(s, m’, c’) • Cache internal transitions as <s, m, c, m’> tuples • No need for Q-learning on these

  10. Idea 2 • Identify weakest precondition P(s) for this internal transition to occur (cf EBL, chunking) • Cache internal transitions as <P, m, c, m’> tuples • Cache size independent of |S|, roughly proportional to size of partial program call graph

  11. The HAMQ-INT Algorithm • Track the set of predicates since last choice point • Save an abstracted rule of internal transition if qualified (τ = 0) in a dictionary ρ • Use the saved rules to shortcircuit the computation of Q values recursively whenever possible

  12. Experimental Result on Taxi

  13. 3 vs 2 Keepaway Comparisons • Option (Stone, 2005): • Each keeper learning separately • Learn a policy over Hold() and Pass(k, v) if ball kickable; otherwise, follow a fixed policy • Intercept() if fastest to the ball; otherwise, GetOpen() • GetOpen() is manually programmed for Option • Concurrent-Option: • Concurrent version of Option • One global Q function is learnt • Random: randomized version of Option • Concurrent-HAMQ • Learn its own version of GetOpen() by calling Stay() and Move(d, v) • Concurrent-HAMQ-INT

  14. Experimental Result on Keepaway

  15. Before and After Initial policy Converged policy

  16. Summary • HAMQ-INT algorithm • Automatically discovers internal transitions • Takes advantage of internal transitions for efficient learning • Outperforms the state of the art significantly on Taxi and RoboCup Keepaway • Future work • Scale up to full RoboCup task • More general integration of model-based and model-free reinforcement learning • More flexible forms of partial program (e.g., temporal logic)

Recommend

SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background

SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background

SDRL: Symbolic Deep Reinforcement Learning SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background Leveraging Symbolic Planning Method Experiment Conclusion Bo Liu and Future Work Auburn

1k views • 20 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3. Policy Gradient and Gradient Estimators 4. Q-prop: Sample Efficient Policy Gradient and an Off-policy Critic 5. Model Based Planning in Discrete

769 views • 53 slides
Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly, K., Zhou, A., Quillen, D., Finn, C., &amp; Levine, S ICML, 2019 Presented by: Egill Ian Gudmundsson Some Terminology On-policy learning: Only one

427 views • 16 slides
Efficient Planning 1 R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction

Efficient Planning 1 R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction

Efficient Planning 1 R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction Tuesday class summary: Planning: any computational process that uses a model to create or improve a policy Dyna framework: 2 R. S. Sutton and A. G.

557 views • 27 slides
Gaussian Processes for Sample Efficient Reinforcement Learning with RMAX-like Exploration Tobias

Gaussian Processes for Sample Efficient Reinforcement Learning with RMAX-like Exploration Tobias

Gaussian Processes for Sample Efficient Reinforcement Learning with RMAX-like Exploration Tobias Jung and Peter Stone Department of Computer Science University of Texas at Austin { tjung,pstone } @cs.utexas.edu Outline: 1. Motivation &amp;

349 views • 18 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak

Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak

Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine Presented by: Samuel Yigzaw 1 OUTLINE Introduction Background Main Contributions Related Work Experiments Conclusion PAGE 2

1.4k views • 26 slides
PEARL Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables Kate

PEARL Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables Kate

PEARL Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables Kate Rakelly*, Aurick Zhou*, Deirdre Quillen, Chelsea Finn, Sergey Levine Hula Beach, Never grow up, The Sled - by artist Matt Spangler,

603 views • 12 slides
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*, Sharad Vikram*, Laura Smith, Pieter Abbeel, Matthew J Johnson, Sergey Levine UC Berkeley, UC San Diego, Google Efficient reinforcement learning from

212 views • 20 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
Harnessing Wake Vortices for Efficient Collective Swimming via Deep Reinforcement Learning

Harnessing Wake Vortices for Efficient Collective Swimming via Deep Reinforcement Learning

Harnessing Wake Vortices for Efficient Collective Swimming via Deep Reinforcement Learning Siddhartha Verma With: Guido Novati and Petros Koumoutsakos CSE lab http://www.cse-lab.ethz.ch Collective Swimming Hydrodynamic benefit of

1.24k views • 27 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone &amp; Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone &amp; Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

682 views • 24 slides
Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still assume a Marko kov decision process (MDP): A se set o of st states s s S A se set o of a actions ( s (per st state) A A mo model

96 views • 5 slides
R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec 03 2008 1 Large State Spaces h When a problem has a large state space we can not longer represent the U or Q functions as explicit tables explicit

436 views • 18 slides
Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning&lt;br/&gt;&lt;br/&gt; 4/25/19, 8*06 PM Reinforcement Learning&lt;br/&gt;&lt;br/&gt; 4/25/19, 8*06 PM Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning? Spring 2019 Created:

371 views • 15 slides
7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views • 56 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi Willem van de Meent Northeastern University [These slides were created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188

145 views • 10 slides
Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning: an Introduction, 2nd Edition: Chapters 6 (6.1 6.5) Outline Reinforcement Learning Reinforcement Learning: the

587 views • 27 slides
Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Machine Learning Supervised Learning Teacher tells learner what to remember Reinforcement Learning

401 views • 29 slides
1 Pure Reinforcement Learning vs. Reinforcement Learning Monte-Carlo Planning No knowledge

1 Pure Reinforcement Learning vs. Reinforcement Learning Monte-Carlo Planning No knowledge

CSE 573: Artificial Intelligence Todo Reinforcement Learning Add simulations from 473 Dan Weld Add UCB bound (cut bolzman &amp; constant epsilon Add snazzy videos (pendulum, zico kolter See http://www See http://www-

462 views • 11 slides

More recommend