Projections for Approximate Policy Iteration Algorithms Riad Akrour - PowerPoint PPT Presentation

Oct 09, 2022 •311 likes •374 views

Projections for Approximate Policy Iteration Algorithms Riad Akrour , Joni Pajarinen, Gerhard Neumann, Jan Peters IAS, TU Darmstadt, Germany ICML19 Entropy Regularization in RL Widespread with actor-critic methods ICML19 Hard vs Soft

Projections for Approximate Policy Iteration Algorithms Riad Akrour , Joni Pajarinen, Gerhard Neumann, Jan Peters IAS, TU Darmstadt, Germany ICML19
Entropy Regularization in RL Widespread with actor-critic methods ICML19
Hard vs Soft Constraints ● Soft constraint (bonus term) Entropy reg. Policy return ● Hard constraint – Harder to optimize, easier to interpret and tune ICML19
Contributions ● Projections hard constraining Shannon entropy of Gaussian or soft-max policies ● Projections that outperform other KL-constrained optimizers used in deep RL ICML19
Results ● Optimizing vs – Deep RL – Projected gradient – Direct policy search ICML19
Results ● Optimizing vs – Deep RL Poster #34 Poster #34 – Projected gradient – Direct policy search ICML19

Recommend

Policy iteration comments Each step of policy iteration is guaranteed to strictly improve the

Policy iteration comments Each step of policy iteration is guaranteed to strictly improve the policy at some state when improvement is possible MDPs cont, Lecture 25 Converge to optimal policy Converge to optimal policy Gives exact

211 views • 4 slides

Trust region policy optimization (TRPO) Value Iteration Value Iteration This is what we

Trust region policy optimization (TRPO) Value Iteration Value Iteration This is what we similar to what Q-Learning does, the main difference being that we we might not know the actual expected reward and instead explore the world and use

435 views • 18 slides

1 Example: Policy Evaluation Policy Evaluation Always Go Right Always Go Forward How do we

Solving MDPs CSE 473: Introduction to Artificial Intelligence Markov Decision Processes II Value Iteration Policy Iteration Reinforcement Learning Steve Tanimoto Based on slides by: Dan Klein and Pieter Abbeel --- University of

433 views • 4 slides

1 Example: Policy Evaluation Policy Evaluation Always Go Right Always Go Forward s How do

Solving MDPs CSE 473: Introduction to Artificial Intelligence Markov Decision Processes II Value Iteration Policy Iteration Reinforcement Learning Based on slides by: Dan Klein and Pieter Abbeel --- University of California, Berkeley

285 views • 4 slides

Introduction to Mobile Robotics The Markov Decision Problem Value Iteration and Policy Iteration

Introduction to Mobile Robotics The Markov Decision Problem Value Iteration and Policy Iteration Wolfram Burgard Cyrill Stachniss Giorgio Grisetti What is the problem? Consider a non-perfect system. Actions are performed with a

899 views • 43 slides

CS 473: Artificial Intelligence MDP Planning: Value Iteration and Policy Iteration Travis Mandel

CS 473: Artificial Intelligence MDP Planning: Value Iteration and Policy Iteration Travis Mandel (subbing for Dan Weld) University of Washington Slides by Dan Klein & Pieter Abbeel / UC Berkeley. (http://ai.berkeley.edu) and by Dan Weld,

436 views • 28 slides

Markov Decision Processes and Exact Solution Methods: Value Iteration Policy Iteration Linear

Markov Decision Processes and Exact Solution Methods: Value Iteration Policy Iteration Linear Programming Pieter Abbeel UC Berkeley EECS Markov Decision Process Assumption: agent gets to observe the state [Drawing from Sutton and Barto,

500 views • 34 slides

Module 7 Policy Iteration CS 886 Sequential Decision Making and Reinforcement Learning

Module 7 Policy Iteration CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Policy Optimization Value iteration Optimize value function Extract induced policy Can we directly optimize the

589 views • 12 slides

Algorithms using real numbers (continued) Noter ch.4 Algorithms using real numbers

Algorithms using real numbers (continued) Noter ch.4 Algorithms using real numbers Representable numbers Rounding/truncation IEEE standard and Java Summation order Newton iteration More iteration

197 views • 18 slides

CS221: Algorithms and Data Structures Recursion and Iteration Alan J. Hu (Borrowing many slides

CS221: Algorithms and Data Structures Recursion and Iteration Alan J. Hu (Borrowing many slides from Steve Wolfman) 1 Learning Goals By the end of this unit, you will be able to Describe the relationship between recursion/iteration

2.43k views • 189 slides

On Conservative Policy Iteration Bruno Scherrer INRIA Lorraine, LORIA ICML 2014 1 / 13

On Conservative Policy Iteration Bruno Scherrer INRIA Lorraine, LORIA ICML 2014 1 / 13 Motivation / Context Large Markov Decision Process A policy space A reference policy On-Policy data from Can we compute a

744 views • 41 slides

Approximate Q-Learning 3-25-16 Exploration policy vs. optimal policy Where do the exploration

Approximate Q-Learning 3-25-16 Exploration policy vs. optimal policy Where do the exploration traces come from? We need some policy for acting in the environment before we understand it. Wed like to get decent rewards while

264 views • 14 slides

New Algorithms for Approximate Minimization of the Difference Between Submodular Functions, with

Background v = f g Procedures for min f g Additional Theoretical Results Experiments Summary New Algorithms for Approximate Minimization of the Difference Between Submodular Functions, with Applications Rishabh Iyer Jeff Bilmes

644 views • 37 slides

Quantum and Classical Algorithms for Approximate Submodular Function Minimization Yassine

Quantum and Classical Algorithms for Approximate Submodular Function Minimization Yassine Hamoudi, Patrick Rebentrost, Ansis Rosmanis, Miklos Santha arXiv: 1907.05378 1. Approximate Submodular Function Minimization 2. Quantum speed-up for

994 views • 98 slides

Quantum and Classical Algorithms for Approximate Submodular Function Minimization Yassine

1.54k views • 113 slides

Approximate Cross-Validation and Dynamic Experiments for Policy Choice Maximilian Kasy

Approximate Cross-Validation and Dynamic Experiments for Policy Choice Approximate Cross-Validation and Dynamic Experiments for Policy Choice Maximilian Kasy Department of Economics, Harvard University April 23, 2018 1 / 23 Approximate

588 views • 23 slides

Approximate Dynamic Programming A. LAZARIC ( SequeL Team @INRIA-Lille ) ENS Cachan - Master 2 MVA

Approximate Dynamic Programming A. LAZARIC ( SequeL Team @INRIA-Lille ) ENS Cachan - Master 2 MVA SequeL INRIA Lille MVA-RL Course Value Iteration: the Idea 1. Let V 0 be any vector in R N 2. At each iteration k = 1 , 2 , . . . , K

1.11k views • 63 slides

Counting and sampling algorithms at low temperature New frontiers in approximate counting STOC

Counting and sampling algorithms at low temperature New frontiers in approximate counting STOC 2020 Will Perkins (UIC) Algorithms and phase transitions When are phase transitions barriers to e ffi cient algorithms? What algorithmic

497 views • 30 slides

Parallel Numerical Algorithms Chapter 5 Eigenvalue Problems Section 5.2 Eigenvalue

Basics Power Iteration QR Iteration Krylov Methods Other Methods Parallel Numerical Algorithms Chapter 5 Eigenvalue Problems Section 5.2 Eigenvalue Computation Michael T. Heath and Edgar Solomonik Department of Computer Science

577 views • 42 slides

Approximate Implicitization and CAD- type intersection algorithms Tor Dokken and Vibeke Skytt

Approximate Implicitization and CAD- type intersection algorithms Tor Dokken and Vibeke Skytt SINTEF ICT, Department of Applied Mathematics This work is partly funded by the European Union through the projects: GAIA II, IST-2001-35512,

437 views • 28 slides

Approximate Search of Regular Expressions Using Bit-Parallel Algorithms Kristo Tammeoja Jaak Vilo

Approximate Search of Regular Expressions Using Bit-Parallel Algorithms Kristo Tammeoja Jaak Vilo Teooriapevad Ruges, 2007 Contents Regular expression (RE) syntax Glushkovs automaton Existing bit-parallel algorithms Exact

817 views • 52 slides

Fast algorithms for sparse principal component analysis based on Rayleigh quotient iteration

Fast algorithms for sparse principal component analysis based on Rayleigh quotient iteration Volodymyr Kuleshov Department of Computer Science Stanford University June 18, 2013 Volodymyr Kuleshov Algorithms for sparse PCA Sparse principal

97 views • 8 slides

Approximate inference: Sampling methods Probabilistic Graphical Models Sharif University of

Approximate inference: Sampling methods Probabilistic Graphical Models Sharif University of Technology Spring 2018 Soleymani Approximate inference Approximate inference techniques Deterministic approximation Variational algorithms

1.95k views • 97 slides

Iteration/loops variety of iteration constructs provided with varying degrees of complexity,

Iteration/loops variety of iteration constructs provided with varying degrees of complexity, well only touch on a subset iteration inherently impure from a functional programming point of view could be implemented purely under

446 views • 6 slides