on conservative policy iteration
play

On Conservative Policy Iteration Bruno Scherrer INRIA Lorraine, - PowerPoint PPT Presentation

Nov 22, 2023 •320 likes •744 views

On Conservative Policy Iteration Bruno Scherrer INRIA Lorraine, LORIA ICML 2014 1 / 13 Motivation / Context Large Markov Decision Process A policy space A reference policy On-Policy data from Can we compute a

  1. On Conservative Policy Iteration Bruno Scherrer INRIA Lorraine, LORIA ICML 2014 1 / 13

  2. Motivation / Context • Large Markov Decision Process • A policy space Π • A reference policy π ∈ Π • On-Policy data from π Can we compute a provably better policy ? • Conservative Policy Iteration (Kakade & Langford, 2002; Kakade, 2003) • When local (gradient) optimization induces a (good) global performance guarantee 2 / 13

  3. Motivation / Context • Large Markov Decision Process • A policy space Π • A reference policy π ∈ Π • On-Policy data from π Can we compute a provably better policy ? • Conservative Policy Iteration (Kakade & Langford, 2002; Kakade, 2003) • When local (gradient) optimization induces a (good) global performance guarantee 2 / 13

  4. Motivation / Context • Large Markov Decision Process • A policy space Π • A reference policy π ∈ Π • On-Policy data from π Can we compute a provably better policy ? • Conservative Policy Iteration (Kakade & Langford, 2002; Kakade, 2003) • When local (gradient) optimization induces a (good) global performance guarantee 2 / 13

  5. Outline 1 Markov Decision Processes 2 Conservative Policy Iteration 3 Practical Issues for a Guaranteed Improvement 3 / 13

  6. Outline 1 Markov Decision Processes 2 Conservative Policy Iteration 3 Practical Issues for a Guaranteed Improvement 4 / 13

  7. Infinite-Horizon Markov Decision Process (Puterman, 1994; Bertsekas & Tsitsiklis, 1996; Sutton & Barto, 1998) Markov Decision Process (MDP): • X is the state space, • A is the action space, • r : X → R is the reward function, ( r t = r ( x t )) • p : X × A → ∆ X is the transition function. ( x t +1 ∼ p ( ·| x t , a t )) Problem: Find a policy π : X → A that maximizes the value v π ( x ) for all x : � ∞ � � � � γ t r t v π ( x ) = E � x 0 = x , {∀ t , a t = π ( x t ) } ( γ ∈ (0 , 1)) � . � t =0 5 / 13

  8. Infinite-Horizon Markov Decision Process (Puterman, 1994; Bertsekas & Tsitsiklis, 1996; Sutton & Barto, 1998) Markov Decision Process (MDP): • X is the state space, • A is the action space, • r : X → R is the reward function, ( r t = r ( x t )) • p : X × A → ∆ X is the transition function. ( x t +1 ∼ p ( ·| x t , a t )) Problem: Find a policy π : X → A that maximizes the value v π ( x ) for all x : � ∞ � � � � γ t r t v π ( x ) = E � x 0 = x , {∀ t , a t = π ( x t ) } ( γ ∈ (0 , 1)) � . � t =0 5 / 13

  9. Notations • For any policy π , v π is the unique solution of the Bellman equation : � ∀ x , v π ( x ) = r ( x ) + γ p ( y | x , π ( x )) v π ( y ) ⇔ v π = T π v π y ∈X v π = ( I − γ P π ) − 1 r . ⇔ v π = r + γ P π v π ⇔ • The optimal value v ∗ is the unique solution of the Bellman optimality equation : � � � ∀ x , v ∗ ( x ) = max r ( x ) + γ p ( y | x , a ) v ∗ ( y ) ⇔ v ∗ = Tv ∗ a ∈A y ∈X ⇔ v ∗ = max T π v ∗ . π • π is a greedy policy w.r.t. v , written π = G v , iff � � � ∀ x , π ( x ) ∈ arg max r ( x ) + γ p ( y | x , a ) v ( y ) ⇔ T π v = Tv . a ∈A y ∈X 6 / 13

  10. Outline 1 Markov Decision Processes 2 Conservative Policy Iteration 3 Practical Issues for a Guaranteed Improvement 7 / 13

  11. Approximate Policy Iteration (Exact) Policy Iteration π k +1 ← G v π k (where v π k = T π k v π k ) • Guaranteed improvement in all states • π is ( ǫ, ν ) -approximately greedy with respect to v , written π = G ǫ ( ν, v ), iff ν T ( Tv − T π v ) = E x ∼ ν { [ Tv ]( x ) − [ T π v ]( x ) } ≤ ǫ. API (Bertsekas & Tsitsiklis, 1996) π k +1 ← G ǫ ( ν, v π k ) • Performance may decrease in all states! 8 / 13

  12. Approximate Policy Iteration (Exact) Policy Iteration π k +1 ← G v π k (where v π k = T π k v π k ) • Guaranteed improvement in all states • π is ( ǫ, ν ) -approximately greedy with respect to v , written π = G ǫ ( ν, v ), iff ν T ( Tv − T π v ) = E x ∼ ν { [ Tv ]( x ) − [ T π v ]( x ) } ≤ ǫ. API (Bertsekas & Tsitsiklis, 1996) π k +1 ← G ǫ ( ν, v π k ) • Performance may decrease in all states! 8 / 13

  13. Approximate Policy Iteration (Exact) Policy Iteration π k +1 ← G v π k (where v π k = T π k v π k ) • Guaranteed improvement in all states • π is ( ǫ, ν ) -approximately greedy with respect to v , written π = G ǫ ( ν, v ), iff ν T ( Tv − T π v ) = E x ∼ ν { [ Tv ]( x ) − [ T π v ]( x ) } ≤ ǫ. API (Bertsekas & Tsitsiklis, 1996) π k +1 ← G ǫ ( ν, v π k ) • Performance may decrease in all states! 8 / 13

  14. Conservative Policy Iteration as a Projected Gradient Ascent Algorithm • π : current policy • π ′ : alternative policy • π α = (1 − α ) π + απ ′ : α -mixture of π and π ′ Taylor expansion of α �→ ν T v π α = E x ∼ ν [ v π α ( x )] around α = 0: ν T ( v π α − v π ) = ν T [( I − γ P π α ) − 1 r − v π ] = ν T ( I − γ P π α ) − 1 ( r − v π + γ P π α v π ) = ν T [( I − γ P π ) − 1 + o ( α )]( T π α v π − T π v π ) = ν T [( I − γ P π ) − 1 + o ( α )] α ( T π ′ v π − T π v π ) α ν,π ( T π ′ v π − T π v π ) + o ( α 2 ) 1 − γ d T = with d T ν,π = (1 − γ ) ν T ( I − γ P π ) − 1 . • The steepest direction is π ′ ∈ G v π . • Choosing π ′ ∈ G ǫ ( d ν,π , v π ) amounts to find an approximately steepest direction. 9 / 13

  15. Conservative Policy Iteration as a Projected Gradient Ascent Algorithm • π : current policy • π ′ : alternative policy • π α = (1 − α ) π + απ ′ : α -mixture of π and π ′ Taylor expansion of α �→ ν T v π α = E x ∼ ν [ v π α ( x )] around α = 0: ν T ( v π α − v π ) = ν T [( I − γ P π α ) − 1 r − v π ] = ν T ( I − γ P π α ) − 1 ( r − v π + γ P π α v π ) = ν T [( I − γ P π ) − 1 + o ( α )]( T π α v π − T π v π ) = ν T [( I − γ P π ) − 1 + o ( α )] α ( T π ′ v π − T π v π ) α ν,π ( T π ′ v π − T π v π ) + o ( α 2 ) 1 − γ d T = with d T ν,π = (1 − γ ) ν T ( I − γ P π ) − 1 . • The steepest direction is π ′ ∈ G v π . • Choosing π ′ ∈ G ǫ ( d ν,π , v π ) amounts to find an approximately steepest direction. 9 / 13

  16. Conservative Policy Iteration as a Projected Gradient Ascent Algorithm • π : current policy • π ′ : alternative policy • π α = (1 − α ) π + απ ′ : α -mixture of π and π ′ Taylor expansion of α �→ ν T v π α = E x ∼ ν [ v π α ( x )] around α = 0: ν T ( v π α − v π ) = ν T [( I − γ P π α ) − 1 r − v π ] = ν T ( I − γ P π α ) − 1 ( r − v π + γ P π α v π ) = ν T [( I − γ P π ) − 1 + o ( α )]( T π α v π − T π v π ) = ν T [( I − γ P π ) − 1 + o ( α )] α ( T π ′ v π − T π v π ) α ν,π ( T π ′ v π − T π v π ) + o ( α 2 ) 1 − γ d T = with d T ν,π = (1 − γ ) ν T ( I − γ P π ) − 1 . • The steepest direction is π ′ ∈ G v π . • Choosing π ′ ∈ G ǫ ( d ν,π , v π ) amounts to find an approximately steepest direction. 9 / 13

  17. Conservative Policy Iteration as a Projected Gradient Ascent Algorithm • π : current policy • π ′ : alternative policy • π α = (1 − α ) π + απ ′ : α -mixture of π and π ′ Taylor expansion of α �→ ν T v π α = E x ∼ ν [ v π α ( x )] around α = 0: ν T ( v π α − v π ) = ν T [( I − γ P π α ) − 1 r − v π ] = ν T ( I − γ P π α ) − 1 ( r − v π + γ P π α v π ) = ν T [( I − γ P π ) − 1 + o ( α )]( T π α v π − T π v π ) = ν T [( I − γ P π ) − 1 + o ( α )] α ( T π ′ v π − T π v π ) α ν,π ( T π ′ v π − T π v π ) + o ( α 2 ) 1 − γ d T = with d T ν,π = (1 − γ ) ν T ( I − γ P π ) − 1 . • The steepest direction is π ′ ∈ G v π . • Choosing π ′ ∈ G ǫ ( d ν,π , v π ) amounts to find an approximately steepest direction. 9 / 13

  18. Conservative Policy Iteration as a Projected Gradient Ascent Algorithm • π : current policy • π ′ : alternative policy • π α = (1 − α ) π + απ ′ : α -mixture of π and π ′ Taylor expansion of α �→ ν T v π α = E x ∼ ν [ v π α ( x )] around α = 0: ν T ( v π α − v π ) = ν T [( I − γ P π α ) − 1 r − v π ] = ν T ( I − γ P π α ) − 1 ( r − v π + γ P π α v π ) = ν T [( I − γ P π ) − 1 + o ( α )]( T π α v π − T π v π ) = ν T [( I − γ P π ) − 1 + o ( α )] α ( T π ′ v π − T π v π ) α ν,π ( T π ′ v π − T π v π ) + o ( α 2 ) 1 − γ d T = with d T ν,π = (1 − γ ) ν T ( I − γ P π ) − 1 . • The steepest direction is π ′ ∈ G v π . • Choosing π ′ ∈ G ǫ ( d ν,π , v π ) amounts to find an approximately steepest direction. 9 / 13

Recommend

Policy iteration comments Each step of policy iteration is guaranteed to strictly improve the

Policy iteration comments Each step of policy iteration is guaranteed to strictly improve the

Policy iteration comments Each step of policy iteration is guaranteed to strictly improve the policy at some state when improvement is possible MDPs cont, Lecture 25 Converge to optimal policy Converge to optimal policy Gives exact

211 views • 4 slides
Trust region policy optimization (TRPO) Value Iteration Value Iteration This is what we

Trust region policy optimization (TRPO) Value Iteration Value Iteration This is what we

Trust region policy optimization (TRPO) Value Iteration Value Iteration This is what we similar to what Q-Learning does, the main difference being that we we might not know the actual expected reward and instead explore the world and use

435 views • 18 slides
1 Example: Policy Evaluation Policy Evaluation Always Go Right Always Go Forward How do we

1 Example: Policy Evaluation Policy Evaluation Always Go Right Always Go Forward How do we

Solving MDPs CSE 473: Introduction to Artificial Intelligence Markov Decision Processes II Value Iteration Policy Iteration Reinforcement Learning Steve Tanimoto Based on slides by: Dan Klein and Pieter Abbeel --- University of

433 views • 4 slides
1 Example: Policy Evaluation Policy Evaluation Always Go Right Always Go Forward s How do

1 Example: Policy Evaluation Policy Evaluation Always Go Right Always Go Forward s How do

Solving MDPs CSE 473: Introduction to Artificial Intelligence Markov Decision Processes II Value Iteration Policy Iteration Reinforcement Learning Based on slides by: Dan Klein and Pieter Abbeel --- University of California, Berkeley

285 views • 4 slides
Introduction to Mobile Robotics The Markov Decision Problem Value Iteration and Policy Iteration

Introduction to Mobile Robotics The Markov Decision Problem Value Iteration and Policy Iteration

Introduction to Mobile Robotics The Markov Decision Problem Value Iteration and Policy Iteration Wolfram Burgard Cyrill Stachniss Giorgio Grisetti What is the problem? Consider a non-perfect system. Actions are performed with a

899 views • 43 slides
CS 473: Artificial Intelligence MDP Planning: Value Iteration and Policy Iteration Travis Mandel

CS 473: Artificial Intelligence MDP Planning: Value Iteration and Policy Iteration Travis Mandel

CS 473: Artificial Intelligence MDP Planning: Value Iteration and Policy Iteration Travis Mandel (subbing for Dan Weld) University of Washington Slides by Dan Klein & Pieter Abbeel / UC Berkeley. (http://ai.berkeley.edu) and by Dan Weld,

436 views • 28 slides
Markov Decision Processes and Exact Solution Methods: Value Iteration Policy Iteration Linear

Markov Decision Processes and Exact Solution Methods: Value Iteration Policy Iteration Linear

Markov Decision Processes and Exact Solution Methods: Value Iteration Policy Iteration Linear Programming Pieter Abbeel UC Berkeley EECS Markov Decision Process Assumption: agent gets to observe the state [Drawing from Sutton and Barto,

500 views • 34 slides
Module 7 Policy Iteration CS 886 Sequential Decision Making and Reinforcement Learning

Module 7 Policy Iteration CS 886 Sequential Decision Making and Reinforcement Learning

Module 7 Policy Iteration CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Policy Optimization Value iteration Optimize value function Extract induced policy Can we directly optimize the

589 views • 12 slides
Class 25: Potential energy and conservation of energy Conservative and non conservative forces

Class 25: Potential energy and conservation of energy Conservative and non conservative forces

Class 25: Potential energy and conservation of energy Conservative and non conservative forces Conservative force Non conservative forces 1. W independent of path. 1. W not only depends on W depends only on initial initial and final

632 views • 7 slides
Iteration/loops variety of iteration constructs provided with varying degrees of complexity,

Iteration/loops variety of iteration constructs provided with varying degrees of complexity,

Iteration/loops variety of iteration constructs provided with varying degrees of complexity, well only touch on a subset iteration inherently impure from a functional programming point of view could be implemented purely under

446 views • 6 slides
Program Analysis with Local Policy Iteration George Karpenkov VERIMAG May 6, 2015 George

Program Analysis with Local Policy Iteration George Karpenkov VERIMAG May 6, 2015 George

Program Analysis with Local Policy Iteration George Karpenkov VERIMAG May 6, 2015 George Karpenkov Program Analysis with Local Policy Iteration 1/41 1 / 41 Outline Algorithm 2/41 Program Analysis with Local Policy Iteration George

1.26k views • 84 slides
Structured Policy Iteration for Linear Quadratic Regulator Youngsuk Park 1 with R. Rossi 2 , Z.

Structured Policy Iteration for Linear Quadratic Regulator Youngsuk Park 1 with R. Rossi 2 , Z.

Structured Policy Iteration for Linear Quadratic Regulator Youngsuk Park 1 with R. Rossi 2 , Z. Wen 3 , G. Wu 2 , and H. Zhao 2 Stanford University 1 , Adobe research 2 , Deepmind 3 July 14, 2020 Introduction reinforcement learning (RL) is

319 views • 18 slides
Matrix Iteration Higher Modes Inverse Iteration Matrix Iteration Giacomo Boffi with Shifts

Matrix Iteration Higher Modes Inverse Iteration Matrix Iteration Giacomo Boffi with Shifts

Matrix Iteration Giacomo Boffi Introduction Fundamental Mode Analysis Second Mode Analysis Matrix Iteration Higher Modes Inverse Iteration Matrix Iteration Giacomo Boffi with Shifts Alternative Dipartimento di Ingegneria Strutturale,

1.06k views • 68 slides
Online Exploration in Least-Squares Policy Iteration Lihong Li, Michael L. Littman, and

Online Exploration in Least-Squares Policy Iteration Lihong Li, Michael L. Littman, and

Online Exploration in Least-Squares Policy Iteration Lihong Li, Michael L. Littman, and Christopher R. Mansley Rutgers Laboratory for Real-Life Reinforcement Learning (RL 3 ) 5/14/2009 AAMAS - Budapest 1 Contributions Reinforcement Learning

850 views • 21 slides
Projections for Approximate Policy Iteration Algorithms Riad Akrour , Joni Pajarinen, Gerhard

Projections for Approximate Policy Iteration Algorithms Riad Akrour , Joni Pajarinen, Gerhard

Projections for Approximate Policy Iteration Algorithms Riad Akrour , Joni Pajarinen, Gerhard Neumann, Jan Peters IAS, TU Darmstadt, Germany ICML19 Entropy Regularization in RL Widespread with actor-critic methods ICML19 Hard vs Soft

374 views • 6 slides
2 +326bps vs In support to the Conservative +25.7% reference index since strategys

2 +326bps vs In support to the Conservative +25.7% reference index since strategys

2 +326bps vs In support to the Conservative +25.7% reference index since strategys financial policy at +23.5% vs year-end 2018 the launch of the Low risk deployment holding level vs year-end 2018 view strategy in 2012 Luxury

761 views • 29 slides
Prediction and Control by Dynamic Programing CS60077: Reinforcement Learning Abir Das IIT

Prediction and Control by Dynamic Programing CS60077: Reinforcement Learning Abir Das IIT

Prediction and Control by Dynamic Programing CS60077: Reinforcement Learning Abir Das IIT Kharagpur Aug 8,9,29,30, Sep 05, 2019 Agenda DP Policy Evaluation Policy Iteration Value Iteration Mathematical Tools DP Extensions Agenda

908 views • 76 slides
Review of conservative vector fields Recall that a vector field F is conservative if there is a

Review of conservative vector fields Recall that a vector field F is conservative if there is a

Review of conservative vector fields Recall that a vector field F is conservative if there is a function f (the potential ) such that F = f . Let D = R 2 { (0 , 0) } , and let F be a vector field on D with continuous first order partial

623 views • 14 slides
CATTLE MARKET ITERATION 01 CIRCULATION PRESENTATION ITERATION 01 INTRODUCTION This document

CATTLE MARKET ITERATION 01 CIRCULATION PRESENTATION ITERATION 01 INTRODUCTION This document

LISKEARD CATTLE MARKET ITERATION 01 CIRCULATION PRESENTATION ITERATION 01 INTRODUCTION This document summarises the extent of the progress made to date on the Liskeard Masterplan Proposal. The proposed masterplan has come as Feasibility a

438 views • 22 slides
15-780: MarkovDecisionProcesses J. Zico Kolter Feburary 29, 2016 1 Outline Introduction

15-780: MarkovDecisionProcesses J. Zico Kolter Feburary 29, 2016 1 Outline Introduction

15-780: MarkovDecisionProcesses J. Zico Kolter Feburary 29, 2016 1 Outline Introduction Formal definition Value iteration Policy iteration Linear programming for MDPs 2 1988 Judea Pearl publishes Probabilistic Reasoning in Intelligent

729 views • 41 slides
Module 9 LAO* CS 886 Sequential Decision Making and Reinforcement Learning University of

Module 9 LAO* CS 886 Sequential Decision Making and Reinforcement Learning University of

Module 9 LAO* CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Large State Space Value Iteration, Policy Iteration and Linear Programming Complexity at least quadratic in || Problem: ||

128 views • 11 slides
CONVERGENCE OF A GENERALIZED MIDPOINT ITERATION JARED ABLE, DANIEL BRADLEY, ALVIN MOON, AND

CONVERGENCE OF A GENERALIZED MIDPOINT ITERATION JARED ABLE, DANIEL BRADLEY, ALVIN MOON, AND

CONVERGENCE OF A GENERALIZED MIDPOINT ITERATION JARED ABLE, DANIEL BRADLEY, ALVIN MOON, AND XINGPING SUN Abstract. We give an analytic proof for the Hausdorff convergence of the midpoint or derived polygon iteration. We generalize this iteration

409 views • 10 slides
Iteration and Debugging Check out Iteration from SVN Loop review Debugging Java programs

Iteration and Debugging Check out Iteration from SVN Loop review Debugging Java programs

Iteration and Debugging Check out Iteration from SVN Loop review Debugging Java programs using Eclipse If you had trouble with Arc2D on Faces, take a look at ArcExample.java and ArcDrawer.java after class. While loop syntax: while (

649 views • 11 slides
Blockly Lists & Iteration CT @ VT Things we are seeing Using lists to represent a data

Blockly Lists & Iteration CT @ VT Things we are seeing Using lists to represent a data

Introduction to Computational Thinking Blockly Lists & Iteration CT @ VT Things we are seeing Using lists to represent a data stream Blockly blocks for iteration Using iteration to process a data stream (i.e., a list)

275 views • 10 slides

More recommend