dac the double actor critic architecture for learning
play

DAC: The Double Actor-Critic Architecture for Learning Options - PowerPoint PPT Presentation

Jun 22, 2023 •167 likes •456 views

DAC: The Double Actor-Critic Architecture for Learning Options NeurIPS 2019 Shangtong Zhang, Shimon Whiteson Presenter: Ehsan Mehralian March 17, 2020 Outline Problem statement Option Critic Double Actor Critic Problem statement

  1. DAC: The Double Actor-Critic Architecture for Learning Options NeurIPS 2019 Shangtong Zhang, Shimon Whiteson Presenter: Ehsan Mehralian March 17, 2020

  2. Outline • Problem statement • Option Critic • Double Actor Critic

  3. Problem statement • Temporal abstraction is a key component in RL: • Better exploration • Faster learning • Better generalization • Transfer learning • MDP + Temporal Abstract actions = SMDP • SMDP algorithms are data ine ffi cient —> The option framework (Sutton et al., 1999) • Rises two problems: • Learning options • Learning a master policy 3

  4. Previous works • Based on finding subgoals: • Di ffi cult to scale up • Can be as expensive as the entire task • Using value-based methods: • Can’t cope with large action spaces • Policy based methods have better convergence properties with function approximation 4

  5. The Option Critic framework (PL Bacon et.al, 2017) • Blurs the line between discovering options and learning options • The first scalable end-to-end approach • No slow down within a single task • Faster convergence in transfer learning 5

  6. <latexit sha1_base64="pYEcPR864IoyROfGrRImFH96CHg=">ACu3icbVHbtNAEF2bWwmXBnjkZURU4SAT2VxEXyIVEBIvSOGStlI2sdabtb3Ua5vdcUXk+id5429YJxGUtiOtdPacGc3MmbjKpcEg+O24167fuHlr53bvzt1793f7Dx4emrLWXEx5mZf6OGZG5LIQU5SYi+NKC6biXBzFJ+87/ehUaCPL4huKjFXLC1kIjlDS0X9X3ufgIoftTwF2nz13/pfPOzoQ8Tzw9W8NJFHhmCD5NmVIMaNujlVw8gzEwnVLFfkYNxUwga6nOSvCs+pcZwhVpYH+YxXHzoe04VgM1UkH1L8Hr2GE7o6ZWUYPjsF0VBYJrtrNFIsGn4etjvDMRIEP51vOo/4gGAXrgMsg3IB2cbEmkCXJa+VKJDnzJhZGFQ4b5hGyXNh162NqBg/YamYWVgwJcy8WXvfwp5lpCU2r4CYc2er2iYMmalYpvZbW0uah15lTarMdmfN7KoahQF3zRK6hywhO6QsJRacMxXFjCupZ0VeMY042jP3bMmhBdXvgwOX4zCl6PXn18NDt5t7dghj8kT4pGQvCEH5COZkCnhzr6zcFInc8cud7+7+SbVdbY1j8h/4dZ/AHcW1q0=</latexit> <latexit sha1_base64="/x6PvheGy3sArpYDHIY2QZBqjE=">ADM3icbVJNixMxGM6MX2v96urRy4ulOJWxzOgueimsiqAHoat2d6Fph0yacPOl0lmsczmP3nxj3gQxIMiXv0PZqZVa7svBJ48z/udhHnMpfK8L5Z97vyFi5e2LjeuXL12/UZz+aBzApB2YBmcSaOQiJZzFM2UFzF7CgXjCRhzA7D42eVfnjChORZ+lbNczZKyDTlEadEGSrYtl62XwFm7wp+Arh84z5xXzvSJR0X+o68e1rDfuA5sgMunpIkIYB1o41zPr4HPSBihPyPixmjFNBazDByj/mU6cIYbmJuahWH5XFcKQBLnkD+z8Gp2I4eYlkQal6vh6XmKeRmutFG+NS3fe1CNSpDwXVkuOGjgShJY4J0JxEkPVlV65Lmv0oM4uYWJy51ybIWvCjLgWn/N6KZs59v9EGjVotryuVxtsAn8JWmhp/aD5CU8yWiQsVTQmUg59L1ejsqpAY6YbuJAsJ/SYTNnQwJQkTI7K+s01tA0zgSgT5qQKanY1oiSJlPMkNJ7VruW6VpFnacNCRY9HJU/zQrGULgpFRQwqg+oDwYQLRlU8N4BQwU2vQGfE7EuZb9YwS/DXR94EBw+6/sPu7v5Oa+/pch1b6Da6gxzko0doD71AfTRA1Ppgfba+Wd/tj/ZX+4f9c+FqW8uYW+g/s3/9Bu9JBrA=</latexit> <latexit sha1_base64="ozDunxBrI7/gQrzaWm7L8O9tGs=">ACG3icbZDLSsNAFIYn3q23qks3g0WsEriBV162bgR6qW10JRyMp2gzNJnJkIJfY93Pgqblwo4kpw4ds4jVlo6w8DH/85hzPn9yPOlHacL2tsfGJyanpmNjc3v7C4lF9eqaowloRWSMhDWfNBUc4CWtFMc1qLJAXhc3rt35wM6td3VCoWBle6F9GgE7A2oyANlYzv32GPXobszvsJZf2kX1RVDZs2bhcVJv3KZabTlFtYdvrgBCAvX4zX3BKTio8Cm4GBZSp3Mx/eK2QxIGmnBQqu46kW4kIDUjnPZzXqxoBOQGOrRuMABVSNJb+vjDeO0cDuU5gUap+7viQSEUj3hm04BuquGawPzv1o91u2DRsKCKNY0ID+L2jHOsSDoHCLSUo07xkAIpn5KyZdkEC0iTNnQnCHTx6F6nbJ3Sntne8WDo+zOGbQGlpHReSifXSITlEZVRBD+gJvaBX69F6t6s95/WMSubWUV/ZH1+A9nFnZU=</latexit> <latexit sha1_base64="FD9wRK489dXj4FzHPCVtNLhkGI=">ADcXicbVLbtNAFJ04PEp4pYUNQqCrRhFOMZFdQLCJVEBIbJBSIG2lTGKNJ+NkVL+YGVdErtf8Hzt+g0/wNhxm5D0SpbOnPs69/p6ScClsu3fNaN+7fqNm1u3Grfv3L13v7m9cyTjVFA2oHEQixOPSBbwiA0UVwE7SQjoRewY+/0Q+E/PmNC8j6puYJG4VkGnGfU6I05W7XfrY/A2bfU34GOPtqvbO+mNIiHQv6pnx2XsK+a5uyAxaekjAkgPNGyd8vAc9IGKQ/LDzbCaMUVyLGYxmNp7yXTgijDQLzXzvOxjXnAkBSx5CMkywCzYTj7EMg3dTPWcfJxhHvlqni9kjDP1wsmFq86la1uw2nKk9fmC0AwnRChOAihk5SvPqkPyvISJrp4wnM9ZUnoGdfyE15uZbPG4UWm9jaWZS5Ka+X2UjlU0hX0hSld1ZOX4jtus2V37dJgEzgVaKHK+m7zF57ENA1ZpGhApBw6dqJGWSGPBixv4FSyhNBTMmVDSMSMjnKyovJoa2ZCfix0F+koGRXMzISjkPR1Z/Cm57ivIq3zDVPlvRxmPklSxiC4a+WkAKobi/GDCBaMqmGtAqOBaK9AZ0ctW+kgbegnO+sib4Gi/67zsvj581Tp4X61jCz1Gu8hEDnqDtAn1EcDRGt/jIfGE+Op8bf+qA713UWoUatyHqD/rP78H3fwF8A=</latexit> Background • MDP M ≡ { S, A, R ( s, a ) , P ( s 0 | s, a ) , P 0 ( s ) , γ } • Goal: ∞ π ∗ = arg max X γ t − 1 r t | s 0 , π θ ] ρ ( π θ ) = arg max E τ ∼ p θ ( τ ) [ θ θ t =1 • Policy Gradient ∂π ( s, a ) ∂ρ X X ∂θ = d π ( s ) Q π ( s, a ) ∂θ s a ∞ X γ t Pr ( s t = s | s 0 , π ) d π ( s ) = 6 t =0

Recommend

Soft Actor-Critic Zikun Chen, Minghan Li Jan. 28, 2020 Soft Actor-Critic: Ofg-Policy Maximum

Soft Actor-Critic Zikun Chen, Minghan Li Jan. 28, 2020 Soft Actor-Critic: Ofg-Policy Maximum

Soft Actor-Critic Zikun Chen, Minghan Li Jan. 28, 2020 Soft Actor-Critic: Ofg-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine Outline Problem: Sample

1.42k views • 62 slides
Actor-Critic Policy Learning in Cooperative Planning Josh Redding, Alborz Geramifard Han-Lim Choi

Actor-Critic Policy Learning in Cooperative Planning Josh Redding, Alborz Geramifard Han-Lim Choi

Actor-Critic Policy Learning in Cooperative Planning Josh Redding, Alborz Geramifard Han-Lim Choi and Jonathan P. How Aerospace Controls Lab, MIT August 22, 2011 Redding et al (ACL) Actor-Critic Cooperative Planning August 22, 2011 1 / 1

611 views • 28 slides
Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model CS330

Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model CS330

Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model CS330 Student Presentation Table of Contents Motivation &amp; problem Method overview Experiments Takeaways Discussion (strengths

653 views • 24 slides
CS 4803 / 7643: Deep Learning Topics: Policy Gradients Actor Critic Ashwin Kalyan

CS 4803 / 7643: Deep Learning Topics: Policy Gradients Actor Critic Ashwin Kalyan

CS 4803 / 7643: Deep Learning Topics: Policy Gradients Actor Critic Ashwin Kalyan Georgia Tech Topics well cover Overview of RL RL vs other forms of learning RL API Applications Framework: Markov

939 views • 51 slides
Actor-Attention-Critic for Multi-Agent Reinforcement Learning Shariq Iqbal and Fei Sha Outline

Actor-Attention-Critic for Multi-Agent Reinforcement Learning Shariq Iqbal and Fei Sha Outline

Actor-Attention-Critic for Multi-Agent Reinforcement Learning Shariq Iqbal and Fei Sha Outline Establish a baseline approach to MARL Demonstrate how recent approaches improve on said baseline through sharing information between

428 views • 11 slides
CS 4803 / 7643: Deep Learning Topics: Policy Gradients Actor Critic Zsolt Kira Georgia

CS 4803 / 7643: Deep Learning Topics: Policy Gradients Actor Critic Zsolt Kira Georgia

CS 4803 / 7643: Deep Learning Topics: Policy Gradients Actor Critic Zsolt Kira Georgia Tech Administrative PS3/HW3 due Tuesday 03/31 PS4/HW4 is optional and due 04/03 There are lots of bonus/Extra credit questions

949 views • 73 slides
Neural Fitted Actor-Critic Matthieu Zimmer Alain Dutech Yann Boniface University of Lorraine,

Neural Fitted Actor-Critic Matthieu Zimmer Alain Dutech Yann Boniface University of Lorraine,

Background Neural Fitted Actor-Critic Future works Neural Fitted Actor-Critic Matthieu Zimmer Alain Dutech Yann Boniface University of Lorraine, LORIA 8 th July 2016 1/18 Background Neural Fitted Actor-Critic Future works Outline

1.21k views • 22 slides
SciForum MOL2NET Efficient Actor-critic Algorithm with Dual Piecewise Model Learning Shan Zhong

SciForum MOL2NET Efficient Actor-critic Algorithm with Dual Piecewise Model Learning Shan Zhong

MOL2NET , 2016 , Vol. 2, J; http://sciforum.net/conference/MOL2NET-02/SUIWML-01 SciForum MOL2NET Efficient Actor-critic Algorithm with Dual Piecewise Model Learning Shan Zhong 1,2,3 , Quan Liu 1,4,5 , Qiming Fu 1,3,5,6 , Peng Zhang 1 , Weisheng

189 views • 4 slides
Soft Actor-Critic: Deep Reinforcement Learning for Robotics Finn Rietz University of Hamburg

Soft Actor-Critic: Deep Reinforcement Learning for Robotics Finn Rietz University of Hamburg

MIN Faculty Department of Informatics Soft Actor-Critic: Deep Reinforcement Learning for Robotics Finn Rietz University of Hamburg Faculty of Mathematics, Informatics and Natural Sciences Department of Informatics Technical Aspects of

742 views • 37 slides
CS 285 Instructor: Sergey Levine UC Berkeley Recap: actor-critic fit a model to estimate

CS 285 Instructor: Sergey Levine UC Berkeley Recap: actor-critic fit a model to estimate

Value Function Methods CS 285 Instructor: Sergey Levine UC Berkeley Recap: actor-critic fit a model to estimate return generate samples (i.e. run the policy) improve the policy Can we omit policy gradient completely? forget policies,

478 views • 28 slides
The Option-Critic Architecture Pierre-Luc Bacon, Jean Harb, Doina Precup Reasoning and Learning

The Option-Critic Architecture Pierre-Luc Bacon, Jean Harb, Doina Precup Reasoning and Learning

The Option-Critic Architecture Pierre-Luc Bacon, Jean Harb, Doina Precup Reasoning and Learning Lab McGill University, Montreal, Canada AAAI 2017 Intelligence: the ability to generalize and adapt efficiently to new and uncertain situations

343 views • 19 slides
Provably Convergent Two- Timescale Off-Policy Actor-Critic with Function Approximation Shangtong

Provably Convergent Two- Timescale Off-Policy Actor-Critic with Function Approximation Shangtong

Provably Convergent Two- Timescale Off-Policy Actor-Critic with Function Approximation Shangtong Zhang 1 , Bo Liu 2 , Hengshuai Yao 3 , Shimon Whiteson 1 1 University of Oxford 2 Auburn University 3 Huawei Preview O ff -policy control under

731 views • 15 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

682 views • 24 slides
ANT Actor Network Theory Relevance Process Architecture Governance Strategies Assemblage

ANT Actor Network Theory Relevance Process Architecture Governance Strategies Assemblage

ANT Actor Network Theory Relevance Process Architecture Governance Strategies Assemblage Theory Actor Network Reflexive Complexity Theory Modernisation Science Plan Basic definition review Work in teams with some silly cases

559 views • 12 slides
Learning agents Performance standard Critic Sensors Learning from Observations feedback

Learning agents Performance standard Critic Sensors Learning from Observations feedback

Learning agents Performance standard Critic Sensors Learning from Observations feedback Environment changes Chapter 18, Sections 13 Learning Performance element element knowledge learning goals experiments Problem generator

179 views • 5 slides
Learning agents Performance standard Critic Sensors Learning from Observations feedback

Learning agents Performance standard Critic Sensors Learning from Observations feedback

Learning agents Performance standard Critic Sensors Learning from Observations feedback Environment changes Chapter 18, Sections 13 Learning Performance element element knowledge learning goals experiments Problem generator

174 views • 6 slides
Data structures wa y x 1 D ASE System System E C r* O r state D Critic Critic E

Data structures wa y x 1 D ASE System System E C r* O r state D Critic Critic E

3/3/2020 Data structures wa y x 1 D ASE System System E C r* O r state D Critic Critic E ACE vector x n R wc Global constants #define MAXINP 256 // Maximum number of input units #define ASE_ETA0 10.0 // ASE default

414 views • 4 slides
The Option-Critic Architecture Author: Pierre-Luc Bacon, Jean Harb, Doina Precup Speaker: Zebin

The Option-Critic Architecture Author: Pierre-Luc Bacon, Jean Harb, Doina Precup Speaker: Zebin

University of Waterloo The Option-Critic Architecture Author: Pierre-Luc Bacon, Jean Harb, Doina Precup Speaker: Zebin KANG June 26, 2018 Content 1 Background Research Problem Markov Decision Process (MDP) Policy Gradient Methods The

830 views • 20 slides
John Osborne (1929 - 1994) John James Osborne was an English playwright, screenwriter, actor and

John Osborne (1929 - 1994) John James Osborne was an English playwright, screenwriter, actor and

John Osborne (1929 - 1994) John James Osborne was an English playwright, screenwriter, actor and critic of The Establishment. In a productive life of more than 40 years, Osborne explored many themes and genres, writing for stage, film and TV. His

309 views • 4 slides
Graph Representation Learning with Graph Convolutional Networks Jure Leskovec Networks: Common

Graph Representation Learning with Graph Convolutional Networks Jure Leskovec Networks: Common

Graph Representation Learning with Graph Convolutional Networks Jure Leskovec Networks: Common Language Movie 1 friend Actor 2 co-worker Actor 1 Mary Peter Actor 4 Movie 3 Tom Movie 2 friend brothers Actor 3 Albert Protein 2

752 views • 57 slides
Living Actor Living Actor Living Actor - Use Cases Living Actor - Use Cases Use Cases

Living Actor Living Actor Living Actor - Use Cases Living Actor - Use Cases Use Cases

Living Actor Living Actor Living Actor - Use Cases Living Actor - Use Cases Use Cases Use Cases Emotions in avatar behavior Emotions in avatar behavior www.cantoche.com Laurent Ach CTO CTO W3C Workshop on Emotion Markup

296 views • 7 slides
Movie &amp; Actor QI, Xiaoxu CHEN, Guanhao JIN, Yue OVERVIEW Goal: build a movie and actor

Movie &amp; Actor QI, Xiaoxu CHEN, Guanhao JIN, Yue OVERVIEW Goal: build a movie and actor

Movie &amp; Actor QI, Xiaoxu CHEN, Guanhao JIN, Yue OVERVIEW Goal: build a movie and actor portal to provide user the data of movie and actor from multiple data source. We can also search for the most popular movie (actor) in a

278 views • 26 slides
Parallel Programming and Heterogeneous Computing D3 - Shared-Nothing: Actors Max Plauth, Sven

Parallel Programming and Heterogeneous Computing D3 - Shared-Nothing: Actors Max Plauth, Sven

Parallel Programming and Heterogeneous Computing D3 - Shared-Nothing: Actors Max Plauth, Sven Khler , Felix Eberhardt, Lukas Wenzel, and Andreas Polze Operating Systems and Middleware Group Actors 1 Actor 0 Actor 1 Actor 3 Actor 4 Actor 2

546 views • 19 slides
Why actor analysis? Actor and network analysis Bert Enserink Network map of linked Network map

Why actor analysis? Actor and network analysis Bert Enserink Network map of linked Network map

Why actor analysis? Actor and network analysis Bert Enserink Network map of linked Network map of websites sexual relations Actor and Network analysis, WHY? Most problems cannot be solved by a single actor; Quality of analysis and

356 views • 12 slides

More recommend