Subgoals in Hierarchical Reinforcement Learning Tianren Tang Tian - PowerPoint PPT Presentation

Sep 27, 2022 •186 likes •330 views

Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning Tianren Tang Tian Tan Shangqi Guo Xiaolin Hu Feng Chen Background Goal-Conditional HRL High policy suffers from non-stationary problem From MARL's

Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning Tianren Tang Tian Tan Shangqi Guo Xiaolin Hu Feng Chen
Background • Goal-Conditional HRL • High policy suffers from non-stationary problem • From MARL's perspective, agent's policy is influenced by other agents • Another Perspective • Usually the action space for high policy is too large, therefore its action which is sub-goal for low policy usually unreachable • Intuitively, action space reduction or action elimination • Drawbacks: • no similiar literature shows how to do space reduction • Reduction or elimination may cause sub-optimal
Intuition • Restrict space into k-step adajecent region
Theoretical Analysis • Shortest Transition Time • For optimal policy 𝜌 ∗ • where 𝜒 −1 : 𝐻 → 𝑇 is a mapping from goal to state s
Theoretical Analysis • k-step adjacent region of s is defined: • Theorem 1: • there is always a surrogate goal 𝑕 ’ ∈ 𝐻 𝐵 that 𝜌 ∗ (𝑏 ∗ |𝑡, 𝑕 ’ ) = 𝜌 ∗ (𝑏 ∗ |𝑡, 𝑕) • Theorem 2: • 𝑕 ’ ∈ 𝐻 𝐵 , 𝑅 ∗ (𝑡, 𝑕 ’ ) = 𝑅 ∗ (𝑡, 𝑕)
Theoretical Optimizations • Original optimization objective where 𝜐 ∗ = (𝑡 0 . . . 𝑡 𝑈𝐿 ), 𝜍 ∗ = (𝑕 0 . . . 𝑕 (𝑈−1)𝐿 ) • Relax above equations:
HRL with Adjacency Constraint • Adjacent Matrix approximation • Contrasitive Loss
Final Optimization Objective • With a learned adjacency network
Algorithm
Experiment Environment • Discrete & Continuous • Result
Abalation Study • Difference: • HRAC-O: HRAC with perfect adajency matrix from environment • NegReward: Relabel reward to negative and bound critic function
Visualization
Summary • Although Intuition is easy, this paper is overall good.

Recommend

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu Topic: Hierarchical RL Presenter: Thophile Gaudin Why Hierarchical

921 views • 22 slides

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement Learning Animesh Garg Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning Richard S. Sutton , Doina

2.18k views • 29 slides

Device Placement Optimization with Reinforcement Learning A Hierarchical Model for Device

Device Placement Optimization with Reinforcement Learning A Hierarchical Model for Device Placement A. Mirhoseini, Hieu Pham, A. Goldie et al November 2019 Problem Background Tensorflow allows user to place operators on different devices to

371 views • 15 slides

Hierarchical Bayesian Methods for Reinforcement Learning David Wingate wingated@mit.edu Joint

Hierarchical Bayesian Methods for Reinforcement Learning David Wingate wingated@mit.edu Joint work with Noah Goodman, Dan Roy, Leslie Kaelbling and Joshua Tenenbaum My Research: Agents Rich sensory data Structured prior knowledge Reasonable

1.04k views • 90 slides

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu: DeepMind Rene Bidart (rbbidart@uwaterloo.ca) CS885 June 22, 2018

653 views • 27 slides

Language as an Abstraction for Hierarchical Deep Reinforcement Learning Paper Authors: Yiding

Language as an Abstraction for Hierarchical Deep Reinforcement Learning Paper Authors: Yiding Jiang, Shixiang Gu, Kevin Murphy, Chelsea Finn Problem Overview Learning a variety of compositional , long horizon skills while being able to

364 views • 34 slides

Regression Idea: dont solve one subgoal by itself, but keep track of all subgoals that must

Regression Idea: dont solve one subgoal by itself, but keep track of all subgoals that must be achieved. Given a set of goals: If they all hold in the initial state, return the empty plan Otherwise, choose an action A that

402 views • 6 slides

Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak

Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine Presented by: Samuel Yigzaw 1 OUTLINE Introduction Background Main Contributions Related Work Experiments Conclusion PAGE 2

1.4k views • 26 slides

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi,

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi, Tuan-Fang Wang, William Yang Wang Publishing year: 2018 Presenter: David Radke CS885 University of Waterloo July, 2020 Overview Problem:

493 views • 24 slides

Hierarchical Reinforcement Learning and Human Behavior Matthew Botvinick Princeton Neuroscience

Hierarchical Reinforcement Learning and Human Behavior Matthew Botvinick Princeton Neuroscience Institute and Department of Psychology Princeton University a a a * v v v Knutson et al., NeuroReport , 2001 Schultz et al.,

845 views • 69 slides

Unsupervised Methods For Subgoal Discovery During Intrinsic Motivation in Model-Free Hierarchical

Unsupervised Methods For Subgoal Discovery During Intrinsic Motivation in Model-Free Hierarchical Reinforcement Learning Jacob Rafati http://rafati.net Co-authored with: David C. Noelle Ph.D. Candidate Electrical Engineering and Computer

648 views • 42 slides

Reinforcement Learning Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu: DeepMind The 34th International Conference on Machine Learning (ICML

719 views • 23 slides

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides

Variational Option Discovery Algorithms Achiam, Edwards, Amodei, Abbeel Topic: Hierarchical

Variational Option Discovery Algorithms Achiam, Edwards, Amodei, Abbeel Topic: Hierarchical Reinforcement Learning Presenter: Harris Chan Overview Motivation : Reward-free option discovery Contributions Background : Universal

561 views • 40 slides

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides

CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov Decision Processes [Put]

CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov Decision Processes [Put] Sec. 11.1-11.3 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Hierarchical RL Hierarchy of goals Reach and actions in Destination

201 views • 8 slides

UT DA Hierarchical and Analytical Hierarchical and Analytical Pla Placement T cement

UT DA Hierarchical and Analytical Hierarchical and Analytical Pla Placement T cement Technique echniques for s for High High- Performa Performance A nce Analog C nalog Circuit ircuits Biying Xu, Shaolan Li, Xiaoqing Xu, Nan Sun, and

506 views • 29 slides

Proposal of a Hierarchical Proposal of a Hierarchical Architecture for Multimodal Architecture for

Proposal of a Hierarchical Proposal of a Hierarchical Architecture for Multimodal Architecture for Multimodal Interactive Systems y Masahiro Araki* 1 Tsuneo Nitta* 2 Kouichi Katsurada* 2 Takuya Nishimoto* 3 Tetsuo Amakasu* 4 Shinnichi Kawamoto* 5

546 views • 35 slides

4303 Ed/Serious Feedback and Reinforcement Feedback and Reinforcement Both of them are

4303 Ed/Serious Feedback and Reinforcement Feedback and Reinforcement Both of them are the most important concepts in learning. Any differences? Feedback Feedback involves providing learners with information about their

710 views • 24 slides

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views • 56 slides

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

683 views • 24 slides

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides

Separation and convexity properties of hierarchical and non hierarchical clustering Patrice

Plan Separation and convexity properties of hierarchical and non hierarchical clustering Patrice Bertrand 1 1 CEREMADE, Universit e Paris-Dauphine, Paris, France Joint work with Jean Diatta 2 2 LIM, Universit e de La R eunion,

401 views • 29 slides

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still assume a Marko kov decision process (MDP): A se set o of st states s s S A se set o of a actions ( s (per st state) A A mo model

96 views • 5 slides