tighter problem dependent regret bounds in reinforcement
play

Tighter Problem-Dependent Regret Bounds in Reinforcement Learning - PowerPoint PPT Presentation

Jan 24, 2024 •122 likes •468 views

Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Bounds Andrea Zanette*, Emma Brunskill zanette@stanford.edu ebrun@cs.stanford.edu Institute for Computational and Mathematical

  1. Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Bounds Andrea Zanette*, Emma Brunskill zanette@stanford.edu ebrun@cs.stanford.edu Institute for Computational and Mathematical Engineering and Department of Computer Science, Stanford University

  2. Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Bounds Andrea Zanette*, Emma Brunskill zanette@stanford.edu ebrun@cs.stanford.edu Institute for Computational and Mathematical Engineering and Department of Computer Science, Stanford University Exploration in RL = Learn quickly how to play near optimally Setting: episodic tabular RL Goal: automatically inherit instance-dependent regret bounds

  3. State of the Art Regret Bounds for Episodic Tabular MDPs

  4. State of the Art Regret Bounds for Episodic Tabular MDPs No Intelligent Exploration ˜ O ( T ) (naive greedy)

  5. State of the Art Regret Bounds for Episodic Tabular MDPs E ffi cient Exploration No Intelligent Exploration ˜ O ( T ) (naive greedy) ˜ O ( HS AT ) (UCRL2, Jaksch 2010)

  6. State of the Art Regret Bounds for Episodic Tabular MDPs E ffi cient Exploration No Intelligent Exploration ˜ ˜ O ( T ) O ( H SAT ) (naive greedy) (Dann 2015) ˜ O ( HS AT ) (UCRL2, Jaksch 2010)

  7. State of the Art Regret Bounds for Episodic Tabular MDPs E ffi cient Exploration No Intelligent Exploration ˜ ˜ O ( T ) O ( H SAT ) (naive greedy) (Dann 2015) ˜ ˜ O ( HS AT ) O ( S HAT ) (Dann 2017) (UCRL2, Jaksch 2010)

  8. State of the Art Regret Bounds for Episodic Tabular MDPs E ffi cient Exploration No Intelligent Exploration ˜ ˜ O ( T ) ˜ O ( H SAT ) O ( HSAT ) (naive greedy) (Dann 2015) (Azar 2017) ˜ ˜ O ( HS AT ) O ( S HAT ) (Dann 2017) (UCRL2, Jaksch 2010)

  9. State of the Art Regret Bounds for Episodic Tabular MDPs Lower Bound E ffi cient Exploration No Intelligent Exploration ˜ ˜ O ( T ) ˜ O ( H SAT ) O ( HSAT ) (naive greedy) (Dann 2015) (Azar 2017) ˜ ˜ O ( HS AT ) O ( S HAT ) Ω ( HSAT ) (Dann 2017) (UCRL2, (Lower Bound) Jaksch 2010)

  10. State of the Art Regret Bounds for Episodic Tabular MDPs Problem Dependent Analysis Lower Bound E ffi cient Exploration No Intelligent Exploration ˜ ˜ O ( T ) ˜ O ( H SAT ) O ( HSAT ) ˜ ℚ ⋆ SAT ) O ( (naive greedy) (Dann 2015) (Azar 2017) ˜ ˜ O ( HS AT ) O ( S HAT ) (Our work) Ω ( HSAT ) (Dann 2017) (UCRL2, (Lower Bound) Jaksch 2010)

  11. Main Result

  12. Main Result ( s , a )

  13. Main Result t t+1 ( s , a )

  14. Main Result t t+1 V ⋆ ( s + 1 ) V ⋆ ( s + 2 ) V ⋆ ( s + ( s , a ) 3 )

  15. Main Result t t+1 V ⋆ ( s + 1 ) V ⋆ ( s + 2 ) V ⋆ ( s + ( s , a ) 3 ) ℚ ⋆ = max s , a Var s + ∼ p ( s , a ) V ⋆ ( s + )

  16. Main Result t t+1 V ⋆ ( s + 1 ) V ⋆ ( s + 2 ) V ⋆ ( s + ( s , a ) 3 ) ℚ ⋆ = max s , a Var s + ∼ p ( s , a ) V ⋆ ( s + ) r 1 + r 2 + … + r H ≤ 𝒣

  17. Main Result t t+1 V ⋆ ( s + 1 ) V ⋆ ( s + 2 ) V ⋆ ( s + ( s , a ) 3 ) ℚ ⋆ = max s , a Var s + ∼ p ( s , a ) V ⋆ ( s + ) r 1 + r 2 + … + r H ≤ 𝒣 min { O ( H SAT ) + [ const ] } 𝒣 2 Main Result : An algorithm with 
 ˜ ˜ ℚ ⋆ SAT ) + [ const ], O ( a (high probability) regret bound:

  18. Main Result t t+1 V ⋆ ( s + 1 ) V ⋆ ( s + 2 ) V ⋆ ( s + ( s , a ) 3 ) ℚ ⋆ = max s , a Var s + ∼ p ( s , a ) V ⋆ ( s + ) r 1 + r 2 + … + r H ≤ 𝒣 min { O ( H SAT ) + [ const ] } 𝒣 2 Main Result : An algorithm with 
 ˜ ˜ ℚ ⋆ SAT ) + [ const ], O ( a (high probability) regret bound: Technique : exploration bonus which is adaptively adjusted as a function of the problem di ffi culty

  19. Long Horizon MDPs

  20. Long Horizon MDPs Standard Setting r ∈ [0,1]

  21. Long Horizon MDPs Standard Setting r ∈ [0,1] H Goal MDP Setting* ∑ r ≥ 0, r t ≤ 1 * this is a more general setting t =1

  22. Long Horizon MDPs Standard Setting r ∈ [0,1] H Goal MDP Setting* ∑ r ≥ 0, r t ≤ 1 * this is a more general setting t =1 COLT Conjecture of Jiang & Agarwal, 2018: Any algorithm must su ff er ~H dependence in terms of sample complexity and regret 
 in the Goal MDP setting

  23. Long Horizon MDPs Standard Setting r ∈ [0,1] H Goal MDP Setting* ∑ r ≥ 0, r t ≤ 1 * this is a more general setting t =1 COLT Conjecture of Jiang & Agarwal, 2018: Any algorithm must su ff er ~H dependence in terms of sample complexity and regret 
 in the Goal MDP setting Our algorithm yields 
 no horizon dependence in the regret bound for the setting 
 of the COLT conjecture without being informed of the setting.

  24. Effect of MDP Stochasticity

  25. Effect of MDP Stochasticity Stochasticity in the Transition Dynamics

  26. Effect of MDP Stochasticity Stochasticity in the Transition Dynamics Deterministic MDP ˜ O ( SAH 2 )

  27. Effect of MDP Stochasticity Stochasticity in the Transition Dynamics Bandit Like Deterministic MDP Structure ˜ O ( SAH 2 ) ˜ O ( SAT + [ . . . ])

  28. Effect of MDP Stochasticity Stochasticity in the Transition Dynamics Bandit Like Deterministic MDP Hard Instances of the Lower Bound Structure ˜ ˜ O ( SAH 2 ) ˜ O ( HSAT + [ . . . ]) O ( SAT + [ . . . ])

  29. Effect of MDP Stochasticity Stochasticity in the Transition Dynamics Bandit Like Deterministic MDP Hard Instances of the Lower Bound Structure ˜ ˜ O ( SAH 2 ) ˜ O ( HSAT + [ . . . ]) O ( SAT + [ . . . ]) Our algorithm matches in dominant terms the best performance for each setting

  30. Related Work (infinite horizon)

  31. Related Work (infinite horizon) In mixing domains: 
 May not improve over worst-case: 
 With domain knowledge: 
 - ( Talebi et al, 2018) 
 -(Maillard et al, 2014) - [REGAL] (Bartlett et al, 2010) 
 - (Ortner, 2018) - [SCAL] (Fruit et al, 2018)

  32. Related Work (infinite horizon) In mixing domains: 
 May not improve over worst-case: 
 With domain knowledge: 
 - ( Talebi et al, 2018) 
 -(Maillard et al, 2014) - [REGAL] (Bartlett et al, 2010) 
 - (Ortner, 2018) - [SCAL] (Fruit et al, 2018) Conclusion - Episodic tabular MDP instance dependent bound without knowledge of the environment - Insights into hardness of RL; provable improvements in many settings of interest

  33. Related Work (infinite horizon) In mixing domains: 
 May not improve over worst-case: 
 With domain knowledge: 
 - ( Talebi et al, 2018) 
 -(Maillard et al, 2014) - [REGAL] (Bartlett et al, 2010) 
 - (Ortner, 2018) - [SCAL] (Fruit et al, 2018) Conclusion - Episodic tabular MDP instance dependent bound without knowledge of the environment - Insights into hardness of RL; provable improvements in many settings of interest near-deterministic MDPs Bandit-structure long horizon MDPs limited range of 
 optimal value function limited variability in value function among successor states

Recommend

Extreme Classification Many modern applications involve a huge number of classes . E.g., image

Extreme Classification Many modern applications involve a huge number of classes . E.g., image

Multi-class SVMs From Tighter Data-Dependent Generalization Bounds to Novel Algorithms Marius Kloft Joint work with Yunwen Lei (CU Hong Kong), Urun Dogan (Microsoft Research), and Alexander Binder (Singapore). Multi-class SVMs From Tighter

358 views • 10 slides
Tighter Bounds on the Inefficiency Ratio of Stable Equilibria in Load Balancing Games Akaki

Tighter Bounds on the Inefficiency Ratio of Stable Equilibria in Load Balancing Games Akaki

Tighter Bounds on the Inefficiency Ratio of Stable Equilibria in Load Balancing Games Akaki Mamageishvili Paolo Penna ETH Zurich Outline Load Balancing Games Inefficiency Ratio of Stable Equilibria Tighter Bounds for IRSE (Our

929 views • 54 slides
Regret bounds for online variational inference Pierre Alquier ACML Nagoya, Nov. 18, 2019

Regret bounds for online variational inference Pierre Alquier ACML Nagoya, Nov. 18, 2019

Regret bounds for online variational inference Pierre Alquier ACML Nagoya, Nov. 18, 2019 Pierre Alquier, RIKEN AIP Regret bounds for online variational inference Co-authors Emtiyaz Khan Badr-Eddine Chrief-Abdellatif Approximate

779 views • 45 slides
Tighter Bounds for the Sum of Irreducible LCP Values Juha Krkkinen 1 Dominik Kempa 1 Marcin

Tighter Bounds for the Sum of Irreducible LCP Values Juha Krkkinen 1 Dominik Kempa 1 Marcin

Tighter Bounds for the Sum of Irreducible LCP Values Juha Krkkinen 1 Dominik Kempa 1 Marcin Pitkowski 2 1 University of Helsinki 2 Nicolaus Copernicus University Outline 1 Cyclic words 2 Irreducible LCP values 3 Upper bound for the sum of

447 views • 33 slides
Tight Bounds on Minimax Regret under Logarithmic Loss via Self-Concordance Blair Bilodeau 1,2,3 ,

Tight Bounds on Minimax Regret under Logarithmic Loss via Self-Concordance Blair Bilodeau 1,2,3 ,

Tight Bounds on Minimax Regret under Logarithmic Loss via Self-Concordance Blair Bilodeau 1,2,3 , Dylan J. Foster 4 , and Daniel M. Roy 1,2,3 Presented at the 2020 International Conference on Machine Learning 1 Department of Statistical Sciences,

907 views • 61 slides
Improved Bounds on Minimax Regret under Logarithmic Loss via Self-Concordance Blair Bilodeau 1 , 2

Improved Bounds on Minimax Regret under Logarithmic Loss via Self-Concordance Blair Bilodeau 1 , 2

Improved Bounds on Minimax Regret under Logarithmic Loss via Self-Concordance Blair Bilodeau 1 , 2 with Dylan J. Foster 3 and Daniel M. Roy 1 , 2 March 11, 2020 1 Department of Statistical Sciences, University of Toronto 2 Vector Institute 3

1.1k views • 95 slides
Regret Bounds for Lifelong Learning Pierre Alquier Groupe de Travail de Machine learning du CMLA

Regret Bounds for Lifelong Learning Pierre Alquier Groupe de Travail de Machine learning du CMLA

Transfer learning, multitask learning, lifelong learning... A strategy for lifelong learning, with regret analysis Open questions Regret Bounds for Lifelong Learning Pierre Alquier Groupe de Travail de Machine learning du CMLA ENS Paris-Saclay

1.33k views • 106 slides
On adaptive regret bounds for non- stochastic bandits Gergely Neu INRIA Lille, SequeL team

On adaptive regret bounds for non- stochastic bandits Gergely Neu INRIA Lille, SequeL team

On adaptive regret bounds for non- stochastic bandits Gergely Neu INRIA Lille, SequeL team Universitat Pompeu Fabra, Barcelona Online learning and bandits Adaptive bounds in online learning Adaptive bounds for bandits Outline

1.22k views • 75 slides
Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior Zi Wang*

Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior Zi Wang*

Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior Zi Wang* Beomjoon Kim* Leslie Pack Kaelbling Dec 5 @ NeurIPS 18 Poster #22 Bayesian optimization x * = argmax Goal: f ( x ) x

346 views • 20 slides
Pessimistic Query Optimization: Tighter Upper Bounds for Intermediate Join Cardinalities Walter

Pessimistic Query Optimization: Tighter Upper Bounds for Intermediate Join Cardinalities Walter

Pessimistic Query Optimization: Tighter Upper Bounds for Intermediate Join Cardinalities Walter Cai Magdalena Balazinska Dan Suciu University of Washington [walter,magda,suciu]@cs.washington.edu July 19th, 2019 Cai, Balazinska, Suciu (UW)

816 views • 56 slides
Module 15 POMDP Bounds CS 886 Sequential Decision Making and Reinforcement Learning University

Module 15 POMDP Bounds CS 886 Sequential Decision Making and Reinforcement Learning University

Module 15 POMDP Bounds CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Bounds POMDP algorithms typically find approximations to optimal value function or optimal policy Need some performance

402 views • 12 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

552 views • 35 slides
Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement

Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement

Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement Learning Quentin Cappart , Emmanuel Goutierre, David Bergman, Louis-Martin Rousseau 1 Research question Bounding mechanisms are critical in the

449 views • 24 slides
CDLG PATHS The notion of REGULAR SOLUTION for a path dependent PDE (1) needs to deal with

CDLG PATHS The notion of REGULAR SOLUTION for a path dependent PDE (1) needs to deal with

Introduction Path dependent second order PDEs Martingale problem for second order elliptic differential operators with path dependent coefficients Time consistent dynamic risk measures P ATH - PEPENDENT PARABOLIC PDE S AND P ATH - DEPENDENT F

832 views • 43 slides
Logarithmic Regret for Learning Linear Quadratic Regulators Efficiently Asaf Cassel Joint work

Logarithmic Regret for Learning Linear Quadratic Regulators Efficiently Asaf Cassel Joint work

Logarithmic Regret for Learning Linear Quadratic Regulators Efficiently Asaf Cassel Joint work with: Alon Cohen, Tomer Koren Reinforcement Learning action u t State x t +1 Cost c t 2 Reinforcement Learning action u t State x t +1 Cost c t

541 views • 31 slides
New Error Bounds for Approximations from Projected Linear Equations H. Yu . Bertsekas

New Error Bounds for Approximations from Projected Linear Equations H. Yu . Bertsekas

Introduction Data-Dependent Error Analysis Applications and Comparisons of Bounds Summary New Error Bounds for Approximations from Projected Linear Equations H. Yu . Bertsekas D. P Department of Computer Science University of

519 views • 18 slides
The Nonstochastic Multi Armed Bandit Problem Part 2 and counting... Shahaf Nacson TAU Nov 15,

The Nonstochastic Multi Armed Bandit Problem Part 2 and counting... Shahaf Nacson TAU Nov 15,

Reminder from last week Goals Lower bounds on the weak regret The Nonstochastic Multi Armed Bandit Problem Part 2 and counting... Shahaf Nacson TAU Nov 15, 2017 Shahaf Nacson The Nonstochastic Multi Armed Bandit Problem Reminder from last

1.05k views • 79 slides
Guaranteed Bounds for Solution of Parameter Dependent System of Equations Andrew Pownuk 1 , Iwona

Guaranteed Bounds for Solution of Parameter Dependent System of Equations Andrew Pownuk 1 , Iwona

Guaranteed Bounds for Solution of Parameter Dependent System of Equations Andrew Pownuk 1 , Iwona Skalna 2 , and Jazmin Quezada 1 1 - The University of Texas at El Paso, El Paso, Texas, USA 2 - AGH University of Science and Technology, Krakow,

418 views • 28 slides
An Estimation Based Allocation Rule with Super-linear Regret and Finite Lock-on Time for

An Estimation Based Allocation Rule with Super-linear Regret and Finite Lock-on Time for

An Estimation Based Allocation Rule with Super-linear Regret and Finite Lock-on Time for Time-dependent Multi-armed Bandit Processes Prokopis C. Prokopiou, Peter E. Caines, and Aditya Mahajan McGill University May 6, 2015 PP, PEC, AM (McGill

434 views • 29 slides
Learning Linear Quadratic Regulators Efficiently with Only Regret T Alon Cohen Joint

Learning Linear Quadratic Regulators Efficiently with Only Regret T Alon Cohen Joint

Learning Linear Quadratic Regulators Efficiently with Only Regret T Alon Cohen Joint work with: Tomer Koren and Yishay Mansour Reinforcement Control Learning Theory Multi-armed Bandits Linear Quadratic Control Agent Environment

763 views • 39 slides
Constant-factor approximation algorithms for the minmax regret problem Juan Pablo Fern andez

Constant-factor approximation algorithms for the minmax regret problem Juan Pablo Fern andez

Constant-factor approximation algorithms for the minmax regret problem Juan Pablo Fern andez G. n Cra 87 N o 30 - 65, Colombia Universidad de Medell e-mail : jpfernandez@udem.edu.co Adviser : Eduardo Conde Universidad de Sevilla, Espa

686 views • 46 slides
Cooperative Inverse Reinforcement Learning Dylan Hadfield-Menell CS237: Reinforcement Learning

Cooperative Inverse Reinforcement Learning Dylan Hadfield-Menell CS237: Reinforcement Learning

Cooperative Inverse Reinforcement Learning Dylan Hadfield-Menell CS237: Reinforcement Learning May 31, 2017 The Value Alignment Problem Example taken from Eliezer Yudkowskys NYU talk The Value Alignment Problem The Value Alignment Problem

1.21k views • 84 slides
R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec 03 2008 1 Large State Spaces h When a problem has a large state space we can not longer represent the U or Q functions as explicit tables explicit

436 views • 18 slides
Data Dependent Priors in PAC-Bayes Bounds John Shawe-Taylor University College London Joint work

Data Dependent Priors in PAC-Bayes Bounds John Shawe-Taylor University College London Joint work

Outline Links PAC-Bayes Analysis Linear Classifiers Data Dependent Priors in PAC-Bayes Bounds John Shawe-Taylor University College London Joint work with Emilio Parrado-Hernndez and Amiran Ambroladze August, 2010 John Shawe-Taylor

1.49k views • 104 slides

More recommend