Reinforcement Learning with a Corrupted Reward Channel Tom Everitt, - PowerPoint PPT Presentation

Mar 16, 2023 •158 likes •275 views

Reinforcement Learning with a Corrupted Reward Channel Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, Shane Legg Australian National University Google DeepMind IJCAI 17 and arXiv Motivation We will need to control

Reinforcement Learning with a Corrupted Reward Channel Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, Shane Legg Australian National University Google DeepMind IJCAI 17 and arXiv
Motivation ● We will need to control Human-Level+ AI ● By identifying problems with various AI-paradigms, we can focus research on – the right paradigms – crucial problems within promising paradigms
The Wireheading Problem ● Future RL agent hijacks reward signal (wireheading) ● CoastRunners agent drives in small circle (misspecified reward function) ● RL agent shortcuts reward sensor (sensory error) ● Cooperative Inverse RL agent misperceives human action (adversarial counterexample)
Formalisation ● Reinforcement Learning is traditionally modeled with Markov Decision Process (MDP): ● This fails to model situations where there is a difference between – True reward – Observed reward ● Can be modeled with Corrupt Reward MDP:
Simplifying assumptions
Good intentions ● Natural optimise true reward using observed reward as evidence ● Theorem: Will still suffer near-maximal regret ● Good intentions is not enough!
Avoiding Over-Optimisation ● Quantilising agent randomly picks a state/policy where reward above threshold ● Theorem: For q corrupt states, exists s.t. has average regret at most ● Avoiding over-optimisation helps!
Richer Information Reward Observation Graphs ● RL: ● Decoupled RL: – States “self-estimate” – Cooperative IRL their reward – Learning values from stories – Learning from Human Preferences
Learning true reward Majority vote Safe state – Cooperative Inverse RL – Learning from Human – Learning values from Preferences stories ● Richer information helps!
Experiments ● AIXIjs: http://aslanides.io/aixijs/demo.html True reward Observed reward
Key Takeaways ● Wireheading: observed reward true reward ● Good intentions is not enough ● Either: – Avoid over-optimisation – Give the agent rich data to learn from (CIRL, stories, human preferences) ● Experiments available online

Recommend

Reinforcement Learning with a Corrupted Reward Channel Tom Everitt, Victoria Krakovna, Laurent

Reinforcement Learning with a Corrupted Reward Channel Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, Shane Legg IJCAI 2017 and arXiv (slides adapted from Tom's IJCAI talk) Motivation Want to give RL agents good incentives

390 views • 20 slides

Reward Shaping in Episodic Reinforcement Learning Marek Grze s Canterbury, UK AAMAS 2017

Reward Shaping in Episodic Reinforcement Learning Marek Grze s Canterbury, UK AAMAS 2017 S ao Paulo, May 812 Motivating Reward Shaping Reinforcement Learning Agent reward state action r t a t s t r t+ 1 Environment s t+ 1 [Sutt

1.41k views • 16 slides

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring environment. Interactions with environment: action state Agent Environment reward Problem: find action policy that maximizes cumulative reward

828 views • 66 slides

Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an

1 Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an action Lecture 2 We get a reward Gillian Hayes Our aim is to learn a policy what action to choose in what state to get maximum reward 11th

641 views • 5 slides

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides

Where's The Reward? Where's The Reward? A Review of Reinforcement Learning for Instructional

Where's The Reward? Where's The Reward? A Review of Reinforcement Learning for Instructional Sequencing Shayan Doroudi 1 2 2 Research Question Research Question Over the past 50 years, how Over the past 50 years, how successful has RL

1.92k views • 163 slides

Interactive Reinforcement Learning Human Generated Reward Presentation for Summer Camp 2015

Interactive Reinforcement Learning Human Generated Reward Presentation for Summer Camp 2015 May 25 2015 Reinforcement Learning Trial and error learning Explore and exploit Sutton and Barto 1988 Represent, predict and control

252 views • 8 slides

Using Natural Language for Reward Shaping in Reinforcement Learning Prasoon Goyal , Scott Niekum

Using Natural Language for Reward Shaping in Reinforcement Learning Prasoon Goyal , Scott Niekum and Raymond J. Mooney The University of Texas at Austin Motivation 2 Motivation In sparse reward settings, random exploration has very

936 views • 47 slides

Learning frameworks Associative reinforcement learning Given input, learn to produce output

Learning frameworks Associative reinforcement learning Given input, learn to produce output (action) that maximizes immediate reward Supervised learning Modified Associative reward-penalty ( A R - P ) Assumes environment specifies correct output

241 views • 5 slides

1 Video of Demo Q-Learning Auto Cliff Grid Exploration vs. Exploitation How to Explore? Video

Reinforcement Learning Reinforcement Learning II We still assume an MDP: A set of states s S A set of actions (per state) A A model T(s,a,s) A reward function R(s,a,s) Still looking for a policy (s) New

358 views • 6 slides

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides

Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent state reward action S t R t A t R t+ 1 Environment S t+ 1

1.07k views • 52 slides

Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David

Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David Silver, Dan Klein, Mausam, Dan Weld Reinforcement Learning At each time step t : Agent executes an action A t Environment emits a reward Rt

1.1k views • 71 slides

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides

Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is

Machine Learning Summer School in Algiers Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is reinforcement learning? a way of programming agents by reward and punishment without needing to specify

1.15k views • 93 slides

CSE-571 Probabilistic Robotics Passive: Policy given, transition model and reward are

Reinforcement Learning Same setting as MDP CSE-571 Probabilistic Robotics Passive: Policy given, transition model and reward are unknown Robot wants to learn value function for the given policy Reinforcement Learning Similar

212 views • 8 slides

Monte Carlo Approaches to Reinforcement Learning Robert Platt (w/ Marcus Gualtieris edits)

Monte Carlo Approaches to Reinforcement Learning Robert Platt (w/ Marcus Gualtieris edits) Northeastern University Model Free Reinforcement Learning Joystick command Agent World Observe screen pixels Reward = game score Goal: learn a

895 views • 47 slides

Lecture 14: Introduction to Reinforcement Learning CS109B Data Science 2 Pavlos Protopapas and

Lecture 14: Introduction to Reinforcement Learning CS109B Data Science 2 Pavlos Protopapas and Mark Glickman Outline What is Reinforcement Learning RL Formalism 1. Reward 2. The agent 3. The environment 4. Actions 5.

1.27k views • 64 slides

Reinforcement Learning l Variation on Supervised Learning l Exact target outputs are not given l

Reinforcement Learning l Variation on Supervised Learning l Exact target outputs are not given l Some variation of reward is given either immediately or after some steps Chess Path Discovery l RL systems learn a mapping from states to

793 views • 36 slides

AIXI: Universal Optimal Sequential Decision Making Marcus Hutter (2005) Reinforcement Learning

AIXI: Universal Optimal Sequential Decision Making Marcus Hutter (2005) Reinforcement Learning State space , Action space , Policy , Reward (, ) Goal: Find policy which maximizes expected cumulative reward.

552 views • 27 slides

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides

Reinforcement Learning Applied to RTS games Leonardo Rosa Amado Felipe Meneguzzi 8 May, 2017

Reinforcement Learning Applied to RTS games Leonardo Rosa Amado Felipe Meneguzzi 8 May, 2017 PUCRS Introduction Introduction Reinforcement learning focuses on maximizing the total reward of an agent through repeated interactions with an

759 views • 28 slides

Learning a Prior over Intent via Meta-Inverse Reinforcement Learning Kelvin Xu, Ellis Ratner, Anca

Learning a Prior over Intent via Meta-Inverse Reinforcement Learning Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, Chelsea Finn University of California, Berkeley Motivation : a well specified reward function remains an important

924 views • 42 slides