Policy Certificates: Towards Accountable Reinforcement Learning - PowerPoint PPT Presentation

Feb 17, 2023 •366 likes •489 views

Policy Certificates: Towards Accountable Reinforcement Learning Christoph Dann, Lihong Li, Wei Wei, Emma Brunskill CMU Google Research Stanford University Minimax-Optimal PAC

Policy Certificates: Towards Accountable Reinforcement Learning Christoph Dann, Lihong Li, Wei Wei, Emma Brunskill CMU Google Research Stanford University
Minimax-Optimal PAC Bounds Key contribution: new algorithm for episodic tabular MDPs with PAC Bound High-prob. Regret Bound S: #states, A: #actions, H: episode length, T: #episodes, ϵ: accuracy
Minimax-Optimal PAC Bounds Key contribution: new algorithm for episodic tabular MDPs with PAC Bound High-prob. Regret Bound First minimax-optimal! (for small ϵ) S: #states, A: #actions, H: episode length, T: #episodes, ϵ: accuracy Prior work: [DLB ‘17] [DB ‘15]
<latexit sha1_base64="30xMli8LGXUeEmLEMSjzuAbUbRA=">ACX3icbVHLTgIxFC3jG1+jroybRmJigiEzg0aXqAtZ+gAlgYF0StGzsP2jpFM5v8BpeuXLnVrR3ARMGbtjk59x721MvElyBZb3mjJnZufmFxaX8srq2rq5sXmrwlhSVqehCGXDI4oJHrA6cBCsEUlGfE+wO69/nuXvnphUPAxqMIiY65P7gPc4JaCpjkla6lFCcnNabTu1FBdxC9gzDPs2ry/O3MSxrAOcrTQZSavtci1Np3T2kRb9HGlSxDdtJ2uadsyCVbKGgaeBPQYFNI7Ljvne6oY09lkAVBClmrYVgZsQCZwKluZbsWIRoX1yz5oaBsRnyk2GV0nxnma6uBdKvQPAQ/Z3RUJ8pQa+p5U+gQc1mcvI/3OU6gupifHQO3ETHkQxsICOpvdigSHEmdm4yWjIAYaECq5fgCmD0QSCvpL8toZe9KHaXDrlOxybk6LFQqY48W0Q7aRfvIRseogqroEtURS/oA32ir9ybsWCsGeZIauTGNVvoTxjb34GntLY=</latexit> Minimax-Optimal PAC Bounds Key contribution: new algorithm for episodic tabular MDPs with PAC Bound High-prob. Regret Bound First minimax-optimal! (for small ϵ) Matches existing + improves for large H S: #states, A: #actions, H: episode length, T: #episodes, ϵ: accuracy Prior work: √ √ H 3 T + S 2 AH 2 SAH 2 T + [AOM ‘17] [DLB ‘17] [DB ‘15]
Motivation: Need for Accountability in Online RL current episode Even with PAC + regret bounds: expected return in next episode during learning unknown
Motivation: Need for Accountability in Online RL How good will my treatment be? Is it the best possible?
Our Proposal: Algorithms output policy certificates before each episode
Algorithms with policy certificates Natural extension of model-based optimistic algorithms 1. UCB on optimal value function 2. Greedy Policy 3. LCB on value function of current policy 4. Output certificate 0
Algorithms with Policy Certificates Natural extension of model-based optimistic algorithms 1. UCB on optimal value function 2. Greedy Policy 3. LCB on value function of current policy 4. Output certificate 0
Symbiosis of Optimism and Certificates Certificates: • Challenge: random • Insight from optimism: at known rate
Symbiosis of Optimism and Certificates Certificates: Optimism: • Challenge: random • Challenge: exploration bonus depends on • Insight from optimism: • Insight from certificates: at known rate bound by
Symbiosis of Optimism and Certificates Certificates: Optimism: • Challenge: random • Challenge: exploration bonus depends on • Insight from optimism: • Insight from certificates: at known rate bound by More accountable algorithms Better exploration bonuses yield through accurate policy certificates minimax-optimal PAC & regret bounds

Recommend

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3. Policy Gradient and Gradient Estimators 4. Q-prop: Sample Efficient Policy Gradient and an Off-policy Critic 5. Model Based Planning in Discrete

769 views • 53 slides

Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

719 views • 40 slides

Reinforcement Learning Policy Op5miza5on Pieter Abbeel UC Berkeley EECS Policy

Reinforcement Learning Policy Op5miza5on Pieter Abbeel UC Berkeley EECS Policy Op>miza>on n Consider control policy parameterized by parameter vector H X max E[ R ( s t ) | ] t =0 n O<en stochas>c policy class

762 views • 36 slides

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement & Imitation) Goal: Find Optimal Policy State/Context s t Agent Imitation Learning: Optimize imitation loss Reinforcement Learning: Optimize

547 views • 53 slides

Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost

Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost using data Solid math for the tuning DNNs support approximations Safety? Explainability? 8 th of November, 2019 S. Gros SARLEM 1 / 1 Reinforcement

311 views • 9 slides

Online Trust and Digital Certificates: The Policy Landscape Stephen Schultze Associate Director

Online Trust and Digital Certificates: The Policy Landscape Stephen Schultze Associate Director Center for Information Technology Policy, Princeton The Stakeholders browsers and opera2ng system vendors cer2ficate

475 views • 30 slides

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

682 views • 24 slides

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides

Module 7 Policy Iteration CS 886 Sequential Decision Making and Reinforcement Learning

Module 7 Policy Iteration CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Policy Optimization Value iteration Optimize value function Extract induced policy Can we directly optimize the

589 views • 12 slides

Who Owns Renewable Energy Certificates: An Exploration of Policy Options and Practice Edward A.

Who Owns Renewable Energy Certificates: An Exploration of Policy Options and Practice Edward A. Holt Ed Holt & Associates, Inc. Ryan H. Wiser and Mark Bolinger Lawrence Berkeley National Laboratory April 2006 Energy Analysis Department

328 views • 16 slides

The Provable E ff ectiveness of Policy Gradient Methods in Reinforcement Learning Sham Kakade

The Provable E ff ectiveness of Policy Gradient Methods in Reinforcement Learning Sham Kakade University of Washington & Microsoft Research (with Alekh Agarwal, Jason Lee, and Gaurav Mahajan) r Policy Optimization in RL [AlphaZero,

1.24k views • 92 slides

Accountable Care Systems Andrew Bland SPG, 26th January 2018 Development of Accountable

Accountable Care Systems Andrew Bland SPG, 26th January 2018 Development of Accountable (Integrated) Care What is meant by accountable care? Organisations in an area delivering better health and care outcomes by managing the total budget

256 views • 15 slides

Jeffrey Levi Professor of Health Policy and Management September 30, 2016 What is

Grantmakers In Health Webinar Jeffrey Levi Professor of Health Policy and Management September 30, 2016 What is accountable health? Accountable health approaches integrate (in varying degrees) the health care and social needs of

796 views • 46 slides

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly, K., Zhou, A., Quillen, D., Finn, C., & Levine, S ICML, 2019 Presented by: Egill Ian Gudmundsson Some Terminology On-policy learning: Only one

427 views • 16 slides

Offline Policy-search in Bayesian Reinforcement Learning Castronovo Michael University of Li`

Offline Policy-search in Bayesian Reinforcement Learning Castronovo Michael University of Li` ege, Belgium Advisor: Damien Ernst 15th March 2017 Contents Introduction Problem Statement Offline Prior-based Policy-search (OPPS)

848 views • 46 slides

Policy Gradients CS60077: Reinforcement Learning Abir Das IIT Kharagpur Nov 09, 10, 2020

Policy Gradients CS60077: Reinforcement Learning Abir Das IIT Kharagpur Nov 09, 10, 2020 Agenda Introduction REINFORCE Bias/Variance Agenda Get started with the policy gradient methods. Get familiar with naive REINFORCE algorithm and

1.11k views • 84 slides

CSE-571 Probabilistic Robotics Passive: Policy given, transition model and reward are

Reinforcement Learning Same setting as MDP CSE-571 Probabilistic Robotics Passive: Policy given, transition model and reward are unknown Robot wants to learn value function for the given policy Reinforcement Learning Similar

212 views • 8 slides

Public Policy and Deep Reinforcement Learning on AWS Emily Webber | Machine Learning Specialist

Public Policy and Deep Reinforcement Learning on AWS Emily Webber | Machine Learning Specialist at Amazon Web Services | To-be-open-sourced research project Public Policy Has Unique Challenges Structural Inefficiency Lack of single goal

387 views • 20 slides

Integrating Oral and General Health: The Role of Accountable Care Organizations Yara Halasa, DDS,

Integrating Oral and General Health: The Role of Accountable Care Organizations Yara Halasa, DDS, MS, MA, PhD(c) The Heller School for Social Policy and Management The Massachusetts Health Policy Forum October 6, 2016 Boston, MA Adult oral

450 views • 16 slides

NCIOM Task Force on Accountable Care Communities January 24, 2018 Kevin Moore VP of Policy

NCIOM Task Force on Accountable Care Communities January 24, 2018 Kevin Moore VP of Policy Health and Human Services United Healthcare Community & State UnitedHealthcare Community & State Serving nearly 6.4 million people**

103 views • 9 slides

Deep Reinforcement Learning through Policy Op7miza7on Pieter Abbeel John Schulman Open AI /

Deep Reinforcement Learning through Policy Op7miza7on Pieter Abbeel John Schulman Open AI / Berkeley AI Research Lab Reinforcement Learning u t [Figure source: SuBon & Barto, 1998] John Schulman & Pieter Abbeel OpenAI + UC

1.21k views • 120 slides

Accountable Care Organization Reporting and Budget Review Test Year Melissa Miles, Health Policy

Accountable Care Organization Reporting and Budget Review Test Year Melissa Miles, Health Policy Project Director Green Mountain Care Board Agenda 113 Statutory Requirements for ACO Budget Review All-Payer ACO Model Agreement 2017

531 views • 15 slides

Who do you Trust? The roles of certificates, certification authorities and the IGTF in Grid

Who do you Trust? The roles of certificates, certification authorities and the IGTF in Grid Computing T h e A m e r i c a s G r i d Policy Management Authority Prof. Vinod Rebello Instituto de Computao Universidade F ederal F

768 views • 18 slides