Kernel-based Reinforcement Learning in Robust Markov Decision - PowerPoint PPT Presentation

Dec 23, 2023 •375 likes •520 views

Kernel-based Reinforcement Learning in Robust Markov Decision Processes Shiau Hong Lim, Arnaud Autef Motivation Robust Markov Decision Process (MDP) framework Tackle model mismatch and parameter uncertainty Previously, for state

Kernel-based Reinforcement Learning in Robust Markov Decision Processes Shiau Hong Lim, Arnaud Autef
Motivation • Robust Markov Decision Process (MDP) framework – Tackle model mismatch and parameter uncertainty – Previously, for state aggregation, performance bound on improved via robust policies: 12/6/2019 Arnaud Autef - ICML 2019 2
Contribution 1. Robust performance bound improvement on extended to the general kernel averager setting 2.Formulation of a practical kernel-based robust algorithm, with empirical results on benchmark tasks 12/6/2019 Arnaud Autef - ICML 2019 3
Kernel-based approach 1.MDP to solve 2.Kernel averager and representative states to approximate the value function: and 12/6/2019 Arnaud Autef - ICML 2019 4
Kernel-based approach 2.Define a non-trivial robust MDP with states = representative states 3.Obtain optimal robust value in 4.Derive in greedy w.r.t , with: 12/6/2019 Arnaud Autef - ICML 2019 5
Theoretical Result Theorem : optimal robust value in , greedy policy w.r.t , optimal value in : ∗ – � � � ∗ – Function approximator limitations � � ∗ Smoothness – � � � � � � 12/6/2019 Arnaud Autef - ICML 2019 6
Practical algorithm 1.Second kernel averager to approximate the MDP model from data 2.Solve with the approximate robust Bellman operator: With Robustness parameter 12/6/2019 Arnaud Autef - ICML 2019 7
Experiments: Acrobot 12/6/2019 Arnaud Autef - ICML 2019 8
Acrobot 12/6/2019 Arnaud Autef - ICML 2019 9
Experiments: Double Pole Balancing 12/6/2019 Arnaud Autef - ICML 2019 10
Double Pole Balancing 12/6/2019 Arnaud Autef - ICML 2019 11
Conclusion • Theoretical performance guarantees for robust kernel-based reinforcement learning in • Significant empirical benefits from robustness, even stronger with model mismatch (real-world settings) 12/6/2019 Arnaud Autef - ICML 2019 12
Thank you! Please come to see our poster tonight Shiau Hong Lim, Arnaud Autef

Recommend

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *,

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *, Yonathan Efroni* and Shie Mannor *equal contribution Poster #272 Action Robust Reinforcement Learning and Applications in Continuous Control Robust MDPs

502 views • 8 slides

Machine Learning and Data Mining Reinforcement Learning Markov Decision Processes Kalev Kask

+ Machine Learning and Data Mining Reinforcement Learning Markov Decision Processes Kalev Kask Overview Intro Markov Decision Processes Reinforcement Learning Sarsa Q-learning Exploration vs Exploitation tradeoff 2

1.81k views • 146 slides

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 is extended by one week, to Wednesday after next Todays Lecture 1. Managing overfitting in model-based RL

686 views • 40 slides

Markov Decision Processes and Reinforcement Learning Marco Chiarandini Department of Mathematics

Lecture 14 Markov Decision Processes and Reinforcement Learning Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Slides by Stuart Russell and Peter Norvig Markov Decision Processes Course

321 views • 29 slides

CS 4803 / 7643: Deep Learning Topic: Reinforcement Learning (RL) Overview Markov

CS 4803 / 7643: Deep Learning Topic: Reinforcement Learning (RL) Overview Markov Decision Processes Zsolt Kira Georgia Tech Administrative PS3/HW3 due March 15 th ! Projects 2 new FB projects up

1.06k views • 85 slides

. . : =L distance functions Possible Euclidean . Manhattan - other .

Midterm the Since Topics based Instance and classification ] learning NN . regression Kernel . mgohssien ... ] ' , margin than Peru " learning , . .dyu, online approaches trick kernel kernel based ^ ,

345 views • 11 slides

Advanced Econometrics 2, Hilary term 2021 Reinforcement learning Maximilian Kasy Department of

Reinforcement learning Advanced Econometrics 2, Hilary term 2021 Reinforcement learning Maximilian Kasy Department of Economics, Oxford University 1 / 21 Reinforcement learning Agenda Markov decision problems: Goal oriented interactions

442 views • 21 slides

Markov Decision Processes CS60077: Reinforcement Learning Abir Das IIT Kharagpur Sep 14 and 15,

Markov Decision Processes CS60077: Reinforcement Learning Abir Das IIT Kharagpur Sep 14 and 15, 2020 Agenda Terminology Markov Decision Process Agenda Understand definitions and notation to be used in the course. Understand definition

1.2k views • 95 slides

Markov Decision Process and Reinforcement Learning Zeqian (Chris) Li Feb 28, 2019 Zeqian

Markov Decision Process and Reinforcement Learning Zeqian (Chris) Li Feb 28, 2019 Zeqian (Chris) Li MDP and RL Feb 28, 2019 1 / 26 Outline 1 Introduction 2 Markov decision process 3 Statistical mechanics of MDP 4 Reinforcement learning 5

503 views • 26 slides

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

682 views • 24 slides

Reproducible, Reusable, and Robust Reinforcement Learning Joelle Pineau Facebook AI Research,

Reproducible, Reusable, and Robust Reinforcement Learning Joelle Pineau Facebook AI Research, Montreal School of Computer Science, McGill University Neural Information Processing Systems (NeurIPS) December 5, 2018 Reproducibility refers to

979 views • 58 slides

Reinforcement Learning CE417: Introduction to Artificial Intelligence Sharif University of

Reinforcement Learning CE417: Introduction to Artificial Intelligence Sharif University of Technology Spring 2019 Soleymani Slides have been adopted from Klein and Abdeel, CS188, UC Berkeley. Reinforcement Learning 2 Recap: MDPs } Markov

906 views • 62 slides

CSE 573: Artificial Intelligence Hanna Hajishirzi Reinforcement Learning II slides adapted from

CSE 573: Artificial Intelligence Hanna Hajishirzi Reinforcement Learning II slides adapted from Dan Klein, Pieter Abbeel ai.berkeley.edu And Dan Weld, Luke Zettelmoyer Reinforcement Learning o Still assume a Markov decision process (MDP): o A

424 views • 40 slides

Examples and Videos of Markov Decision Processes (MDPs) and Reinforcement Learning Artificial

Examples and Videos of Markov Decision Processes (MDPs) and Reinforcement Learning Artificial Intelligence is interaction to achieve a goal Environment action state reward Agent complete agent temporally situated continual

516 views • 36 slides

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Environment dynamics Stochastic processes Markovian assumption

156 views • 14 slides

Logistics Reading AIMA Ch 21 (Reinforcement Learning) Markov Decision Processes Project

Logistics Reading AIMA Ch 21 (Reinforcement Learning) Markov Decision Processes Project 1 due today 2 printouts of report Email Miao with CSE 573 Source code Document in .doc or .pdf Project 2 description on web New

400 views • 8 slides

Markov Decision Processes CS60077: Reinforcement Learning Abir Das IIT Kharagpur July 26, Aug

Markov Decision Processes CS60077: Reinforcement Learning Abir Das IIT Kharagpur July 26, Aug 01, 02, 08, 2019 Agenda Terminology Markov Decision Process Agenda Understand definitions and notation to be used in the course. Understand

1.13k views • 95 slides

Fingerprint Policy Optimisation for Robust Reinforcement Learning Supratik Paul, Michael A.

Poster #50 Fingerprint Policy Optimisation for Robust Reinforcement Learning Supratik Paul, Michael A. Osborne, Shimon Whiteson This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020

385 views • 25 slides

Markov Decision Processes (MDPs) and Reinforcement Learning (RL) Sven Koenig, USC Russell and

12/18/2019 Markov Decision Processes (MDPs) and Reinforcement Learning (RL) Sven Koenig, USC Russell and Norvig, 3 rd Edition, Sections 17.1-17.2 These slides are new and can contain mistakes and typos. Please report them to Sven

449 views • 23 slides

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides

CS885 Reinforcement Learning Lecture 2a: May 4, 2018 Intro to Markov decision processes [SutBar]

CS885 Reinforcement Learning Lecture 2a: May 4, 2018 Intro to Markov decision processes [SutBar] Chap. 3, [Sze] Chap. 2, [RusNor] Sec. 17.1-17.2, 17.4, [Put] Chap. 2, 4, 5 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Markov

509 views • 17 slides

Model Based Reinforcement Learning Oriol Vinyals (DeepMind) @OriolVinyalsML May 2018 Stanford

Model Based Reinforcement Learning Oriol Vinyals (DeepMind) @OriolVinyalsML May 2018 Stanford University The Reinforcement Learning Paradigm OBSERVATIONS GOAL Agent Environment ACTIONS The Reinforcement Learning Paradigm Maximize Return -

1.26k views • 72 slides

MARKOV GAMES A framework for multi-agent reinforcement learning Shen (Sean) Chen Review on

MARKOV GAMES A framework for multi-agent reinforcement learning Shen (Sean) Chen Review on MDPs An MDP is defined by a set of states, S, and actions, A. Transition function, T: S A PD(S), where PD(S) represents discrete prob

287 views • 17 slides