budgeted reinforcement learning in continuous state space
play

Budgeted Reinforcement Learning in Continuous State Space Nicolas - PowerPoint PPT Presentation

Oct 20, 2023 •309 likes •959 views

Budgeted Reinforcement Learning in Continuous State Space Nicolas Carrara 1 , Edouard Leurent 1,2 , Tanguy Urvoy 3 , Romain Laroche 4 , Odalric Maillard 1 , Olivier Pietquin 1,5 1 Inria SequeL, 2 Renault Group, 3 Orange Labs, 4 Microsoft Montr

  1. Budgeted Reinforcement Learning in Continuous State Space Nicolas Carrara 1 , Edouard Leurent 1,2 , Tanguy Urvoy 3 , Romain Laroche 4 , Odalric Maillard 1 , Olivier Pietquin 1,5 1 Inria SequeL, 2 Renault Group, 3 Orange Labs, 4 Microsoft Montr´ eal, 5 Google Research, Brain Team

  2. Contents 01.. Motivation and Setting 02.. Budgeted Dynamic Programming 03.. Budgeted Reinforcement Learning 04.. Experiments 2 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  3. 01 Motivation and Setting 3 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  4. Learning to act Optimal Decision-Making Which action a t should we choose in state s t to maximise a cumulative reward R ? � ∞ � � γ t R ( s t , a t ) max E a t ∼ π ( a t | s t ) π t = 0 4 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  5. Learning to act Optimal Decision-Making Which action a t should we choose in state s t to maximise a cumulative reward R ? � ∞ � � γ t R ( s t , a t ) max E a t ∼ π ( a t | s t ) π t = 0 � A very general formulation � Widely used in the industry 4 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  6. Learning to act Optimal Decision-Making Which action a t should we choose in state s t to maximise a cumulative reward R ? � ∞ � � γ t R ( s t , a t ) max E a t ∼ π ( a t | s t ) π t = 0 � A very general formulation ✗ Not widely used in the industry 4 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  7. Learning to act Optimal Decision-Making Which action a t should we choose in state s t to maximise a cumulative reward R ? � ∞ � � γ t R ( s t , a t ) max E a t ∼ π ( a t | s t ) π t = 0 � A very general formulation ✗ Not widely used in the industry > Sample efficiency > Trial and error > Unpredictable behaviour 4 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  8. Limitation of Reinforcement Learning Reinforcement learning relies on a single reward function R 5 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  9. Limitation of Reinforcement Learning Reinforcement learning relies on a single reward function R � A convenient formulation, but; 5 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  10. Limitation of Reinforcement Learning Reinforcement learning relies on a single reward function R � A convenient formulation, but; ✗ R is not always easy to design. 5 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  11. Limitation of Reinforcement Learning Reinforcement learning relies on a single reward function R � A convenient formulation, but; ✗ R is not always easy to design. Conflicting Objectives Complex tasks require multiple contradictory aspects. Typically: Task completion vs Safety 5 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  12. Limitation of Reinforcement Learning Reinforcement learning relies on a single reward function R � A convenient formulation, but; ✗ R is not always easy to design. Conflicting Objectives Complex tasks require multiple contradictory aspects. Typically: Task completion vs Safety For example... 5 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  13. Example problems with conflicts Dialogue systems A slot-filling problem: the agent fills a form by asking the user each slot. It can either: • ask to answer using voice (safe/slow); • ask to answer with a numeric pad (unsafe/fast). 6 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  14. Example problems with conflicts Dialogue systems A slot-filling problem: the agent fills a form by asking the user each slot. It can either: • ask to answer using voice (safe/slow); • ask to answer with a numeric pad (unsafe/fast). Autonomous Driving The agent is driving on a two-way road with a car in front of it, • it can stay behind (safe/slow); • it can overtake (unsafe/fast). 6 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  15. Limitation of Reinforcement Learning Reinforcement learning relies on a single reward function R � A convenient formulation, but; ✗ R is not always easy to design. Conflicting Objectives Complex tasks require multiple contradictory aspects. Typically: Task completion vs Safety For example... For a fixed reward function R , no control over the Task Completion trade-off Safety π ∗ is only guaranteed to lie on a Pareto-optimal curve Π ∗ 7 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  16. The Pareto-optimal curve Task Completion 𝐻 1 = ∑𝛿 𝑢 𝑆 1 𝑢 Pareto-optimal curve Π ∗ ∑𝛿 𝑢 𝑆 𝑢 (𝑆 1 , 𝑆 2 ) argmax 𝜌 Safety 𝐻 2 = ∑𝛿 𝑢 𝑆 2 𝑢 8 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  17. From maximal safety to minimal risk Task Completion 𝐻 𝑠 Pareto-optimal curve Π ∗ ∑𝛿 𝑢 𝑆 𝑢 (𝑆 𝑠 , −𝑆 𝑑 ) argmax 𝜌 Risk 𝐻 𝑑 9 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  18. The optimal policy can move freely along Π ∗ Task Completion 𝐻 𝑠 Pareto-optimal curve Π ∗ ∑𝛿 𝑢 𝑆 𝑢 (𝑆 𝑠 , −𝑆 𝑑 ) argmax 𝜌 𝜌 ∗ Risk 𝐻 𝑑 10 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  19. How to choose a desired trade-off Task Completion 𝐻 𝑠 Pareto-optimal curve Π ∗ ∑𝛿 𝑢 𝑆 𝑠 𝑢 argmax 𝜌 𝑢 < 𝛾 𝑡. 𝑢. ∑𝛿 𝑢 𝑆 𝑑 𝜌 ∗ Risk 𝐻 𝑑 𝛾 11 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  20. Constrained Reinforcement Learning Markov Decision Process An MDP is a tuple ( S , A , P , R r , γ ) with: • Rewards R r ∈ R S×A Objective Maximise rewards E [ � ∞ t = 0 γ t R r ( s t , a t ) | s 0 = s ] max π ∈M ( A ) S 12 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  21. Constrained Reinforcement Learning Constrained Markov Decision Process A CMDP is a tuple ( S , A , P , R r , R c , γ, β ) with: • Rewards R r ∈ R S×A • Costs R c ∈ R S×A • Budget β Objective Maximise rewards while keeping costs under a fixed budget E [ � ∞ t = 0 γ t R r ( s t , a t ) | s 0 = s ] max π ∈M ( A ) S E [ � ∞ t = 0 γ t R c ( s t , a t ) | s 0 = s ] ≤ β s.t. 12 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  22. We want to learn Π ∗ rather than π ∗ β Task Completion 𝐻 𝑠 Pareto-optimal curve Π ∗ ∑𝛿 𝑢 𝑆 𝑠 𝑢 argmax 𝜌 𝑢 < 𝛾 𝑡. 𝑢. ∑𝛿 𝑢 𝑆 𝑑 𝜌 ∗ Risk 𝐻 𝑑 𝛾 13 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  23. We want to learn Π ∗ rather than π ∗ β Task Completion 𝐻 𝑠 Pareto-optimal curve Π ∗ ∑𝛿 𝑢 𝑆 𝑠 𝑢 argmax 𝜌 𝑢 < 𝛾 𝑡. 𝑢. ∑𝛿 𝑢 𝑆 𝑑 𝜌 ∗ Risk 𝐻 𝑑 𝛾 13 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  24. Budgeted Reinforcement Learning Budgeted Markov Decision Process A BMDP is a tuple ( S , A , P , R r , R c , γ, B ) with: • Rewards R r ∈ R S×A • Costs R c ∈ R S×A • Budget space B Objective Maximise rewards while keeping costs under an adjustable budget. ∀ β ∈ B , E [ � ∞ t = 0 γ t R r ( s t , a t ) | s 0 = s , β 0 = β ] max π ∈M ( A×B ) S×B E [ � ∞ t = 0 γ t R c ( s t , a t ) | s 0 = s , β 0 = β ] ≤ β s.t. 14 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  25. Problem formulation Budgeted policies π • Take a budget β as an additional input • Output a next budget β ′ → ( a , β ′ ) • π : ( s , β ) � �� � � �� � s a Augment the spaces with the budget β 15 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  26. Augmented Setting Definition (Augmented spaces) • States S = S × B . • Actions A = A × B . • Dynamics P � s ′ ∼ P ( s ′ | s , a ) state ( s , β ) , action ( a , β a ) → next state β ′ = β a Definition (Augmented signals) 1. Rewards R = ( R r , R c ) 2. Returns G π = ( G π c ) def = � ∞ t = 0 γ t R ( s t , a t ) r , G π = E [ G π | s 0 = s ] c ) def 3. Value V π ( s ) = ( V π r , V π c ) def = E [ G π | s 0 = s , a 0 = a ] 4. Q-Value Q π ( s , a ) = ( Q π r , Q π 16 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  27. 02 Budgeted Dynamic Programming 17 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  28. Policy Evaluation Proposition (Budgeted Bellman Expectation) The Bellman Expectation equations are preserved � V π ( s ) = π ( a | s ) Q π ( s , a ) a ∈A � s ′ � � � � s , a V π ( s ′ ) Q π ( s , a ) = R ( s , a ) + γ P s ′ ∈S 18 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

  29. Policy Evaluation Proposition (Budgeted Bellman Expectation) The Bellman Expectation equations are preserved � V π ( s ) = π ( a | s ) Q π ( s , a ) a ∈A � s ′ � � � � s , a V π ( s ′ ) Q π ( s , a ) = R ( s , a ) + γ P s ′ ∈S Proposition (Contraction) The Bellman Expectation Operator T π is a γ -contraction. � � T π Q ( s , a ) def P ( s ′ | s , a ) π ( a ′ | s ′ ) Q ( s ′ , a ′ ) = R ( s , a ) + γ s ′ ∈S a ′ ∈A � We can evaluate a budgeted policy π 18 -Budgeted Reinforcement Learning- Carrara N., Leurent E.

Recommend

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec 03 2008 1 Large State Spaces h When a problem has a large state space we can not longer represent the U or Q functions as explicit tables explicit

436 views • 18 slides
Reinforcement Learning for Continuous State and Action Spaces Gradient Methods 1 MACHINE LEARNING

Reinforcement Learning for Continuous State and Action Spaces Gradient Methods 1 MACHINE LEARNING

MACHINE LEARNING 2012 MACHINE LEARNING TECHNIQUES AND APPLICATIONS Reinforcement Learning for Continuous State and Action Spaces Gradient Methods 1 MACHINE LEARNING 2012 Reinforcement Learning (RL) Supervised Learning Unsupervised

777 views • 51 slides
Bayesian Reinforcement Learning in Continuous POMDPs Stphane Ross 1 , Brahim Chaib-draa 2 and

Bayesian Reinforcement Learning in Continuous POMDPs Stphane Ross 1 , Brahim Chaib-draa 2 and

Motivation Bayesian Reinforcement Learning Experiments and Applications Conclusion Bayesian Reinforcement Learning in Continuous POMDPs Stphane Ross 1 , Brahim Chaib-draa 2 and Joelle Pineau 1 1 School of Computer Science, McGill University,

478 views • 18 slides
Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *,

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *,

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *, Yonathan Efroni* and Shie Mannor *equal contribution Poster #272 Action Robust Reinforcement Learning and Applications in Continuous Control Robust MDPs

502 views • 8 slides
Reinforcement Learning in Continuous Environments 64.425 Integrated Seminar: Intelligent Robotics

Reinforcement Learning in Continuous Environments 64.425 Integrated Seminar: Intelligent Robotics

MIN Faculty Department of Informatics University of Hamburg Continuous Reinforcement Learning Reinforcement Learning in Continuous Environments 64.425 Integrated Seminar: Intelligent Robotics Oke Martensen University of Hamburg Faculty of

325 views • 16 slides
Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an

Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an

1 Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an action Lecture 2 We get a reward Gillian Hayes Our aim is to learn a policy what action to choose in what state to get maximum reward 11th

641 views • 5 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still assume a Marko kov decision process (MDP): A se set o of st states s s S A se set o of a actions ( s (per st state) A A mo model

96 views • 5 slides
LunarLander-v2 using Deep Reinforcement Learning A project developed for Autonomous Agents

LunarLander-v2 using Deep Reinforcement Learning A project developed for Autonomous Agents

LunarLander-v2 using Deep Reinforcement Learning A project developed for Autonomous Agents Course PLH513 Portokalakis Petros February 2020 Simple Game 8-Dimensional state space 4 actions per state +100 points for landing

301 views • 10 slides
Continuous Control With Deep Reinforcement Learning Timothy P. Lillicrap , Jonathan J. Hunt

Continuous Control With Deep Reinforcement Learning Timothy P. Lillicrap , Jonathan J. Hunt

Continuous Control With Deep Reinforcement Learning Timothy P. Lillicrap , Jonathan J. Hunt , Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver &amp; Daan Wierstra Presenters: Anqi (Joyce) Yang Jonah Philion Jan 21

1.64k views • 101 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Reinforcement Learning in Configurable Continuous Environments Alberto Maria Metelli, Emanuele

Reinforcement Learning in Configurable Continuous Environments Alberto Maria Metelli, Emanuele

Reinforcement Learning in Configurable Continuous Environments Alberto Maria Metelli, Emanuele Ghelfi and Marcello Restelli 36th International Conference on Machine Learning 13th June 2019 1 Non-Configurable Environments 1 Configurable

596 views • 12 slides
1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides
Guidelines for Action Space Definition in Reinforcement Learning-based Traffic Signal Control

Guidelines for Action Space Definition in Reinforcement Learning-based Traffic Signal Control

Guidelines for Action Space Definition in Reinforcement Learning-based Traffic Signal Control Systems Maxime Trca, Julian Garbiso, Dominique Barth October 15, 2020 Outline I - Reinforcement Learning Applied to Traffic Signal Control II -

189 views • 16 slides
Discrete and Continuous Reinforcement Learning (not part of exam material) 1 1 ADVANCED

Discrete and Continuous Reinforcement Learning (not part of exam material) 1 1 ADVANCED

ADVANCED MACHINE LEARNING ADVANCED MACHINE LEARNING Brief Overview of Discrete and Continuous Reinforcement Learning (not part of exam material) 1 1 ADVANCED MACHINE LEARNING Forms of Learning Supervised learning where the algorithm

1.27k views • 71 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
DeepMDP Learning Latent Space Continuous Models for Representation Learning Carles Gelada,

DeepMDP Learning Latent Space Continuous Models for Representation Learning Carles Gelada,

DeepMDP Learning Latent Space Continuous Models for Representation Learning Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, Marc G. Bellemare Simple Representations for RL 2 12 DeepMDP Latent Space Model: Neural networks MDP:

573 views • 17 slides
Symbolic Regression for Reinforcement Learning and Dynamic System Modeling Robert Babuka 1

Symbolic Regression for Reinforcement Learning and Dynamic System Modeling Robert Babuka 1

Symbolic Regression for Reinforcement Learning and Dynamic System Modeling Robert Babuka 1 Research interests Clustering for building locally linear models Reinforcement learning for continuous dynamic systems Neural

604 views • 31 slides
1 Video of Demo Q-Learning Auto Cliff Grid Exploration vs. Exploitation How to Explore? Video

1 Video of Demo Q-Learning Auto Cliff Grid Exploration vs. Exploitation How to Explore? Video

Reinforcement Learning Reinforcement Learning II We still assume an MDP: A set of states s S A set of actions (per state) A A model T(s,a,s) A reward function R(s,a,s) Still looking for a policy (s) New

358 views • 6 slides
Module 9 LAO* CS 886 Sequential Decision Making and Reinforcement Learning University of

Module 9 LAO* CS 886 Sequential Decision Making and Reinforcement Learning University of

Module 9 LAO* CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Large State Space Value Iteration, Policy Iteration and Linear Programming Complexity at least quadratic in || Problem: ||

128 views • 11 slides
Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent state reward action S t R t A t R t+ 1 Environment S t+ 1

1.07k views • 52 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring environment. Interactions with environment: action state Agent Environment reward Problem: find action policy that maximizes cumulative reward

828 views • 66 slides
Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides
AIXI: Universal Optimal Sequential Decision Making Marcus Hutter (2005) Reinforcement Learning

AIXI: Universal Optimal Sequential Decision Making Marcus Hutter (2005) Reinforcement Learning

AIXI: Universal Optimal Sequential Decision Making Marcus Hutter (2005) Reinforcement Learning State space , Action space , Policy , Reward (, ) Goal: Find policy which maximizes expected cumulative reward.

552 views • 27 slides

More recommend