Fingerprint Policy Optimisation for Robust Reinforcement Learning - PowerPoint PPT Presentation

Poster #50 Fingerprint Policy Optimisation for Robust Reinforcement Learning Supratik Paul, Michael A. Osborne, Shimon Whiteson This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreements \#637713)

Motivation 2

Motivation • Environment variable (EV) • E.g. wind conditions • Controllable during learning but not during execution 2

Motivation • Environment variable (EV) • E.g. wind conditions • Controllable during learning but not during execution • Objective: Find 𝜌 ∗ = 𝑏𝑠𝑕𝑛𝑏𝑦 𝜌 𝐾 𝜌 = 𝑏𝑠𝑕𝑛𝑏𝑦 𝜌 𝔽 𝐹𝑊~𝑞(𝐹𝑊) [𝑆 𝜌 ] 2

Motivation • Environment variable (EV) • E.g. wind conditions • Controllable during learning but not during execution • Objective: Find 𝜌 ∗ = 𝑏𝑠𝑕𝑛𝑏𝑦 𝜌 𝐾 𝜌 = 𝑏𝑠𝑕𝑛𝑏𝑦 𝜌 𝔽 𝐹𝑊~𝑞(𝐹𝑊) [𝑆 𝜌 ] • Need to account for rare events • E.g. rare wind conditions leading to a crash 2

Naïve application of f policy gradients 3

Naïve application of f policy gradients Trajectories ~ 𝜌 3

Naïve application of f policy gradients Rare Events Trajectories ~ 𝜌 4

Naïve application of f policy gradients Rare Events Trajectories ~ 𝜌 • Monte Carlo estimate of the Policy Gradient has very high variance ⟹ Doomed to failure 4

Fingerprint Policy Optimisation (F (FPO) 5

Fingerprint Policy Optimisation (F (FPO) At each iteration, select parameters 𝜔 of 𝑟 𝜔 (𝐹𝑊) such that it maximises one-step expected return 5

Fingerprint Policy Optimisation (F (FPO) 6

Fingerprint Policy Optimisation (F (FPO) • 𝜌 ′ = 𝜌 + α𝛼𝐾 𝜌 • 𝐾 𝜌 ′ = f(𝜌, 𝜔) 6

Fingerprint Policy Optimisation (F (FPO) • 𝜌 ′ = 𝜌 + α𝛼𝐾 𝜌 • 𝐾 𝜌 ′ = f(𝜌, 𝜔) • Model 𝐾 𝜌 ′ as a Gaussian Process with inputs (𝜌, 𝜔) • Use Bayesian Optimisation to select 𝜔|𝜌 = argmax 𝜔 f(𝜌, 𝜔) 6

Fingerprint Policy Optimisation (F (FPO) • 𝜌 ′ = 𝜌 + α𝛼𝐾 𝜌 • 𝐾 𝜌 ′ = f(𝜌, 𝜔) • Model 𝐾 𝜌 ′ as a Gaussian Process with inputs (𝜌, 𝜔) • Use Bayesian Optimisation to select 𝜔|𝜌 = argmax 𝜔 f(𝜌, 𝜔) Low dimensional representation 𝜌 is high dimensional “Fingerprint” 6

Policy fi fingerprints 7

Policy fi fingerprints • Disambiguation, not accurate representation 7

Policy fi fingerprints • Disambiguation, not accurate representation • State/Action fingerprints: Gaussians fitted to the stationary state/action distribution induced by 𝜌 7

Policy fi fingerprints • Disambiguation, not accurate representation • State/Action fingerprints: Gaussians fitted to the stationary state/action distribution induced by 𝜌 • Gross simplification, but good at disambiguating between policies 7

Results Half Cheetah Ant • Velocity target = 2 with probability • Reward proportional to velocity 98% and ‘normal’ reward • 5% chance that velocity > 2 leads to • Velocity target = 4 with probability 2% joint damage with large negative with significantly high reward reward 8

Poster #50 Fingerprint Policy Optimisation for Robust Reinforcement Learning Supratik Paul, Michael A. Osborne, Shimon Whiteson This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreements \#637713)

Fingerprint Policy Optimisation for Robust Reinforcement Learning - PowerPoint PPT Presentation

Poster #50 Fingerprint Policy Optimisation for Robust Reinforcement Learning Supratik Paul, Michael A. Osborne, Shimon Whiteson This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *,

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Kernel-based Reinforcement Learning in Robust Markov Decision Processes Shiau Hong Lim, Arnaud

Reproducible, Reusable, and Robust Reinforcement Learning Joelle Pineau Facebook AI Research,

WE MAKE INNOVATION HAPPEN Pry-ID The cable fingerprint Pry-ID is the fingerprint for your

Reinforcement Learning Policy Op5miza5on Pieter Abbeel UC Berkeley EECS Policy

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost

A Whorlwind Tour A Guide To Expert Fingerprint Evidence Presented by Philip Gilhooley

DactyMatch Green Bit Green Bit Fingerprint Recognition Recognition Fingerprint SDK v.2.2

COSEC DOOR FMX High Performance Door Controller with Multispectral Fingerprint Reader Demanding

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

Optimisation Constraint Problems Combinatorial Optimisation Modelling (in MiniZinc) Solving

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning BUPT Pengda Qin ,

Module 7 Policy Iteration CS 886 Sequential Decision Making and Reinforcement Learning

The Provable E ff ectiveness of Policy Gradient Methods in Reinforcement Learning Sham Kakade

Automated and Accurate Geometry Extraction and Shape Optimisation of 3D Topology Optimisation

SEO (Search Engine Optimisation) May 2015 What is SEO? Optimisation of your website that will

Fi Fingerprint ngerprint Clas Classification sification Ma Maor or Sh Shar arf Fi Fing

Pressure Optimisation Introduction Why carry out Pressure Optimisation How and Who

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Fingerprint Policy Optimisation for Robust Reinforcement Learning - PowerPoint PPT Presentation

Poster #50 Fingerprint Policy Optimisation for Robust Reinforcement Learning Supratik Paul, Michael A. Osborne, Shimon Whiteson This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *,

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Kernel-based Reinforcement Learning in Robust Markov Decision Processes Shiau Hong Lim, Arnaud

Reproducible, Reusable, and Robust Reinforcement Learning Joelle Pineau Facebook AI Research,

WE MAKE INNOVATION HAPPEN Pry-ID The cable fingerprint Pry-ID is the fingerprint for your

Reinforcement Learning Policy Op5miza5on Pieter Abbeel UC Berkeley EECS Policy

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &amp;

Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost

A Whorlwind Tour A Guide To Expert Fingerprint Evidence Presented by Philip Gilhooley

DactyMatch Green Bit Green Bit Fingerprint Recognition Recognition Fingerprint SDK v.2.2

COSEC DOOR FMX High Performance Door Controller with Multispectral Fingerprint Reader Demanding

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

Optimisation Constraint Problems Combinatorial Optimisation Modelling (in MiniZinc) Solving

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning BUPT Pengda Qin ,

Module 7 Policy Iteration CS 886 Sequential Decision Making and Reinforcement Learning

The Provable E ff ectiveness of Policy Gradient Methods in Reinforcement Learning Sham Kakade

Automated and Accurate Geometry Extraction and Shape Optimisation of 3D Topology Optimisation

SEO (Search Engine Optimisation) May 2015 What is SEO? Optimisation of your website that will

Fi Fingerprint ngerprint Clas Classification sification Ma Maor or Sh Shar arf Fi Fing

Pressure Optimisation Introduction Why carry out Pressure Optimisation How and Who

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &