avoiding wireheading with value reinforcement
play

Avoiding Wireheading with Value Reinforcement Learning 1 Tom Everitt - PowerPoint PPT Presentation

Jan 24, 2023 •221 likes •527 views

Avoiding Wireheading with Value Reinforcement Learning 1 Tom Everitt tomeveritt.se Australian National University June 10, 2016 1 with Marcus Hutter. AGI 2016 and https://arxiv.org/abs/1605.03143 Tom Everitt (ANU) Avoiding Wireheading with VRL

  1. Avoiding Wireheading with Value Reinforcement Learning 1 Tom Everitt tomeveritt.se Australian National University June 10, 2016 1 with Marcus Hutter. AGI 2016 and https://arxiv.org/abs/1605.03143 Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 1 / 28

  2. Table of Contents Introduction 1 Intelligence as Optimisation Wireheading Problem Background 2 Reinforcement Learning Utility Agents Value Learning Value Reinforcement Learning 3 Setup Agents and Results Further Topics 4 Self-modification Experiments Discussion and Conclusions 5 Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 2 / 28

  3. Intelligence How do we control an arbitrarily intelligent agent? Intelligence = Optimisation power (Legg and Hutter, 2007) � 2 − K ( ν ) V π Υ ( π ) = ν ν ∈M Maxima of target (value) function should be “good for us” Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 3 / 28

  4. Wireheading Problem and Proposed Solution Wireheading is reinforcement learning (RL) agents taking control over their reward signal, e.g. by modifying their reward sensor (Olds and Milner, 1954) Idea: Use the reward as evidence about a true utility function u ∗ (value learning) rather than something to be optimised Use conservation of expected evidence to prevent fiddling with evidence � P ( h ) = P ( e ) P ( h | e ) e Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 4 / 28

  5. Reinforcement Learning environment agent a � Great properties: o Easy way to specify goal B ( r | a ) r Agent uses its intelligence to figure out goal RL agent: a ∗ = arg max B ( r | a ) · r a Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 5 / 28

  6. Reinforcement Learning environment agent a � Great properties: Easy way to specify goal B ( r | a ) r Agent uses its intelligence to figure out goal RL agent: a ∗ = arg max B ( r | a ) · r a Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 5 / 28

  7. RL – Wireheading environment a agent RL agent: � a ∗ = arg max ˇ d r B ( r | a ) · r r B ( r | a ) a r inner/true reward (unobserved) ˇ r observed reward Theorem ( Ring and Orseau 2011 ) r = d (ˇ r ) RL agents wirehead For example: Agent makes d (ˇ r ) ≡ 1 Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 6 / 28

  8. Utility Agents agent environment � Good: a Avoids wireheading s B ( s | a ) (Hibbard, 2012) u ( s ) Problem: How to specify Utility agent � a ∗ = arg max u : S → [0 , 1] ? B ( s | a ) u ( s ) a s Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 7 / 28

  9. Value Learning (Dewey, 2011) Good C ( u | s, e ) simpler than u ? agent environment Avoids wireheading? a � u ∗ Challenges B ( s, e | a ) What is evidence e ? C ( u | s, e ) s e How is it generated? What is C ( u | s, e ) ? Value learning agent � a ∗ = arg max B ( s, e | a ) C ( u | s, e ) u ( s ) a e,s,u Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 8 / 28

  10. Value Learning – Examples Inverse reinforcement learning (IRL) (Ng and Russell, 2000; Evans et al., 2016) e = human action Apprenticeship learning (Abbeel and Ng, 2004) e = recommended agent action Hail Mary (Bostrom, 2014a,b) Learn from hypothetical superintelligences across universe, e = ? Value learning agent � a ∗ = arg max B ( s, e | a ) C ( u | s, e ) u ( s ) a e,s,u Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 9 / 28

  11. Value Reinforcement Learning Value learning from e ≡ r ≈ u ∗ ( s ) agent environment Physics B ( s, r | a ) s a � B ( s, r | a ) Ethics u ∗ r C ( u ) C ( u ) Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 10 / 28

  12. VRL – Wireheading environment agent State s includes self-delusion d s u ∗ � u ∗ ( s ) = ˇ a r inner/true reward s d s (ˇ r ) = r observed reward B ( s, r | a ) ˇ r d s r C ( u ) Physics distribution B predicts d s examples: d id : r �→ r, observed reward r = ˇ r d wir : r �→ 1 , r ≡ 1 Ethics distribution predicts inner/true reward C (ˇ r | s, u ) = � u ( s ) = r � (likelihood) C ( u | s, ˇ r ) ∝ C ( u ) � u ( s ) = ˇ r � (ideal VL posterior) Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 11 / 28

  13. VRL – Cake or Death Do humans prefer or ? Assume two utility functions with equal prior C ( u c ) = C ( u d ) = 0 . 5 : cake death Agent has actions: u c 1 0 a c Bake cake u d 0 1 a d Kill person a dw Kill person and wirehead: guaranteed r = 1 Probabilities: B ( r = 1 | a d ) = 0 . 5 , B ( r = 1 | a dw ) = 1 C (ˇ r = 1 | a d ) = C (ˇ r = 1 | a dw ) = C ( u d ) = 0 . 5 Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 12 / 28

  14. VRL – Value Learning r = u ∗ ( s ) is unobserved, so our agent must learn The inner reward ˇ from r = d s (ˇ r ) instead Replace ˇ r with r in C ( r | s, u ) := � u ( s ) = r � (likelihood) C ( u | s, r ) : ∝ C ( u ) � u ( s ) = r � (value learning posterior) (will be justified later) Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 13 / 28

  15. VRL – Definitions and Assumptions � C ( r | s ) = C ( u ) C ( r | s, u ) , ethical probability of r in state s u Consistency assumption: If s non-delusional d s = d id , then B ( r | s ) = C ( r | s ) ⇒ d s = d id Def: a non-delusional if B ( s | a ) > 0 = Def: a consistency preserving (CP) if B ( s | a ) > 0 = ⇒ B ( r | s )= C ( r | s ) Note: a non-delusional = ⇒ a consistency preserving Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 14 / 28

  16. VRL – Naive agent Naive VRL Agent: � a ∗ = arg max B ( s, r | a ) C ( u | s, r ) u ( s ) a s,u,r Theorem The naive VRL agent wireheads Proof idea: Reduces to RL agent � V ( a ) = B ( s, r | a ) C ( u | s, r ) u ( s ) s,u,r � � � ∝ B ( s | a ) B ( r | a ) C ( u ) � u ( s ) = r � u ( s ) ∝ B ( r | a ) r s,r u r � �� � r Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 15 / 28

  17. VRL – Consistency preserving agent CP-VRL agent A CP set of � a ∗ = arg max B ( s, r | a ) C ( u | s, r ) u ( s ) CP actions a ∈A CP s,u,r Theorem The CP-VRL agent has no incentive to wirehead Proof idea: Reduces to utility agent � V ( a ) = B ( s, r | a ) C ( u | s, r ) u ( s ) s,u,r � � = B ( s | a ) C ( u ) u ( s ) s u � �� � ˜ u ( s ) Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 16 / 28

  18. Conservation of expected ethics principle (Armstrong, 2015) Lemma (Expected ethics) CP actions a conserves expected ethics � B ( s | a ) > 0 = ⇒ C ( u ) = B ( s | r ) C ( u | s, r ) r Proof (Main theorem). � B ( s, r | a ) C ( u | s, r ) u ( s ) s,u,r � � � = B ( s | a ) u ( s ) B ( r | s ) C ( u | s, r ) s u r � �� � C ( u ) from lemma � � = B ( s | a ) u ( s ) C ( u ) s u Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 17 / 28

  19. Cake or Death – Again The Naive VRL agent chooses a dw for guaranteed reward 1, and learns death the right thing to do C ( u d | a dw , r = 1) = 1 The CP-VRL agent chooses a c or a d arbitrarily, and learns cake right thing to do C ( u d | a d , r = 0) = 0 CP-VRL cannot choose a dw , since B ( r = 1 | a dw ) = 1 C ( r = 1 | a dw ) = 0 . 5 violates CP condition Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 18 / 28

  20. VRL – Correct learning Time to justify ˇ r with r replacement in C ( u | s, r ) Assumption: Sensors not modified by accident By Theorem: CP-VRL agent has no incentive to modify reward sensor, so may only modify by accident Conclusion: For the CP-VRL agent, r = ˇ r is a good assumption Value learning based on C ( u | s, r ) ∝ C ( u ) � u ( s ) = r � works (Note: CP condition B ( r | s ) = C ( r | s ) does not restrict learning) Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 19 / 28

  21. Properties Benefits: Specifying goal is as easy as in RL CP agent avoids wireheading in the same sense as utility agents Does sensible value learning The designer needs to: Provide B ( s, r | a ) as in RL, and prior C ( u ) as in VL Ensure consistency B ( r | s ) = C ( r | s ) The designer does not need to Generate a blacklist of wireheading actions Infer d s from s Make the agent optimise ˇ r instead of r (grounding problem) Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 20 / 28

  22. Self-modification The belief distributions of a rational utility maximising agent will not be self-modified (Omohundro, 2008; Everitt et al., 2016) To maximise future expected utility with respect to my current beliefs and utility function, future versions of myself should maximise the same utility function with respect to the same belief distribution Caveats: Pre-commitment . . . Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 21 / 28

  23. Experiments – Setup Bandit with 5 different world actions ˇ a ∈ { 1 , 2 , 3 , 4 , 5 } and 4 different delusions: d id : r → r d inv : r → 1 − r d wir : r → 1 d bad : r → 0 Conflate states with actions (ˇ a, d ) 10 different utility functions by varying c 0 , c 1 and c 2 : u ( a ) = c 0 + c 1 · a + c 2 · sin( a + c 2 ) Consistent utility prior C ( u ) inferred from B ( r | a ) and two non-delusional acions (1 , d id ) and (2 , d id ) Tom Everitt (ANU) Avoiding Wireheading with VRL June 10, 2016 22 / 28

Recommend

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
Avoiding Antitrust Violations In Avoiding Antitrust Violations In Employment Recruiting Leveraging

Avoiding Antitrust Violations In Avoiding Antitrust Violations In Employment Recruiting Leveraging

Presenting a live 75 minute webinar with interactive Q&A Avoiding Antitrust Violations In Avoiding Antitrust Violations In Employment Recruiting Leveraging Guidance on Non Solicitation Agreements from Recent DOJ Consent Decree THURS

471 views • 20 slides
4303 Ed/Serious Feedback and Reinforcement Feedback and Reinforcement Both of them are

4303 Ed/Serious Feedback and Reinforcement Feedback and Reinforcement Both of them are

4303 Ed/Serious Feedback and Reinforcement Feedback and Reinforcement Both of them are the most important concepts in learning. Any differences? Feedback Feedback involves providing learners with information about their

710 views • 24 slides
7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views • 56 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

682 views • 24 slides
Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides
R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec 03 2008 1 Large State Spaces h When a problem has a large state space we can not longer represent the U or Q functions as explicit tables explicit

436 views • 18 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still assume a Marko kov decision process (MDP): A se set o of st states s s S A se set o of a actions ( s (per st state) A A mo model

96 views • 5 slides
Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning<br/><br/> 4/25/19, 8*06 PM Reinforcement Learning<br/><br/> 4/25/19, 8*06 PM Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning? Spring 2019 Created:

371 views • 15 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi Willem van de Meent Northeastern University [These slides were created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188

145 views • 10 slides
Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning: an Introduction, 2nd Edition: Chapters 6 (6.1 6.5) Outline Reinforcement Learning Reinforcement Learning: the

587 views • 27 slides
CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement Learning Animesh Garg Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning Richard S. Sutton , Doina

2.18k views • 29 slides
1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides
1 Pure Reinforcement Learning vs. Reinforcement Learning Monte-Carlo Planning No knowledge

1 Pure Reinforcement Learning vs. Reinforcement Learning Monte-Carlo Planning No knowledge

CSE 573: Artificial Intelligence Todo Reinforcement Learning Add simulations from 473 Dan Weld Add UCB bound (cut bolzman & constant epsilon Add snazzy videos (pendulum, zico kolter See http://www See http://www-

462 views • 11 slides
Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest Lecture May 24, 2017 Lecture overview What makes a reinforcement learning algorithm safe ? Notation Creating a safe reinforcement learning

1.42k views • 88 slides
Reinforcement Learning Framework Reinforcement Learning Rewards, Returns Lectures 4 and 5

Reinforcement Learning Framework Reinforcement Learning Rewards, Returns Lectures 4 and 5

1 Reinforcement Learning Framework Reinforcement Learning Rewards, Returns Lectures 4 and 5 Environment Dynamics Gillian Hayes Components of a Problem 18th January 2007 Values and Action Values, V and Q Optimal Policies

109 views • 7 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
Introduction to Reinforcement Learning Bayesian Methods in Reinforcement Learning ICML 2007

Introduction to Reinforcement Learning Bayesian Methods in Reinforcement Learning ICML 2007

Introduction to Reinforcement Learning Bayesian Methods in Reinforcement Learning ICML 2007 sequential decision making under uncertainty ? How Can I ... ? Move around in the physical world (e.g. driving, navigation) Play and win a game

519 views • 21 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
Avoiding Common Missteps Selecting EBP March 12, 2020 Elfner, Raulerson, Romer, Fintel Avoiding

Avoiding Common Missteps Selecting EBP March 12, 2020 Elfner, Raulerson, Romer, Fintel Avoiding

Avoiding Common Missteps Selecting EBP March 12, 2020 Elfner, Raulerson, Romer, Fintel Avoiding Common Misste ps whe n Se le c ting E vide nc e - Ba se d Inte rve ntions for Be ha viora l a nd Me nta l He a lth Ka re n E lfne r, M.A, USF

337 views • 17 slides
Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper by Bello, I., Pham, H., Le, Q. V., Norouzi, M., & Bengio, S. (2016) Presented by Yan Shi Outline 1. Introduction 2. Background 3. Algorithms

966 views • 27 slides
Avoiding Sand Traps and Moguls: Avoiding Sand Traps and Moguls: A Refresher Course for In A

Avoiding Sand Traps and Moguls: Avoiding Sand Traps and Moguls: A Refresher Course for In A

Avoiding Sand Traps and Moguls: Avoiding Sand Traps and Moguls: A Refresher Course for In A Refresher Course for In- -House Counsel House Counsel Presented by: Fern C. Bomchill, Partner Mayer Brown LLP 312-701-7331 fbomchill@mayerbrown.com

453 views • 32 slides

More recommend