learned impatience dispersed reinforcement and time
play

Learned Impatience? Dispersed Reinforcement and Time Discounting - PowerPoint PPT Presentation

Jul 21, 2023 •21 likes •471 views

Learned Impatience? Dispersed Reinforcement and Time Discounting David Poensgen (Goethe University Frankfurt) February 22, 2019 Sloan-Nomis Workshop on the Cognitive Foundations of Economic Behavior Motivation 1. Individuals learn from

  1. Learned Impatience? Dispersed Reinforcement and Time Discounting David Poensgen (Goethe University Frankfurt) February 22, 2019 Sloan-Nomis Workshop on the Cognitive Foundations of Economic Behavior

  2. Motivation 1. Individuals learn from consequences of past actions. 2. Actions often have a series of consequences: some follow soon, some later. 3. How does this ordering affect learning? Plausibly: Easiest to learn from soonest consequences. 4. Then: Immediate consequences will be over-weighted. Behavior biased towards impatience. May help explain why myopic behavior is so widespread and persistent. 1

  3. Motivation 1. Individuals learn from consequences of past actions. 2. Actions often have a series of consequences: some follow soon, some later. 3. How does this ordering affect learning? Plausibly: Easiest to learn from soonest consequences. 4. Then: Immediate consequences will be over-weighted. Behavior biased towards impatience. May help explain why myopic behavior is so widespread and persistent. 1

  4. 2

  5. 2

  6. 2

  7. Motivation 1. Individuals learn from consequences of past actions. 2. Actions often have a series of consequences: some follow soon, some later. 3. How does this ordering affect learning? Plausibly: Easiest to learn from soonest consequences. 4. Then: Immediate consequences will be over-weighted. Behavior biased towards impatience. May help explain why myopic behavior is so widespread and persistent. 3

  8. Motivation 1. Individuals learn from consequences of past actions. 2. Actions often have a series of consequences: some follow soon, some later. 3. How does this ordering affect learning? Plausibly: Easiest to learn from soonest consequences. 4. Then: Immediate consequences will be over-weighted. Behavior biased towards impatience. May help explain why myopic behavior is so widespread and persistent. 3

  9. Motivation 1. Individuals learn from consequences of past actions. 2. Actions often have a series of consequences: some follow soon, some later. 3. How does this ordering affect learning? Plausibly: Easiest to learn from soonest consequences. 4. Then: Immediate consequences will be over-weighted. Behavior biased towards impatience. May help explain why myopic behavior is so widespread and persistent. 3

  10. 4

  11. Motivation 1. Individuals learn from consequences of past actions. 2. Actions often have a series of consequences: some follow soon, some later. 3. How does this ordering affect learning? Plausibly: Easiest to learn from soonest consequences. 4. Then: Immediate consequences will be over-weighted. Behavior biased towards impatience. May help explain why myopic behavior is so widespread and persistent. 5

  12. Background • Decreasing effectiveness of reinforcement with delay (e.g. Mazur 2002). • Typically not connected to time discounting, but speed of learning. • Explained via accumulation of noise by Commons, Woodford et al. (1982, 1991). • Feedback delay modulates neural circuitries involved in learning (Foerde/Shohamy 2011, Foerde et al. 2013, Arbel et al. 2017). • Associative learning tasks; singular feedback. Performance not affected. • Gabaix & Laibson (2017) also link time discounting and information frictions. • Formally applicable here; different interpretation on source of noise. • Melioration theory: Behavior guided by immediate, not overall reinforcement rate (Herrnstein et al.). • Important experimental paradigm: “Harvard game” (Review: Prelec 2014). • Critique by Sims et al. (2013): Bayesian algorithms need 1000s of trials for solution. Melioration as rational response to task complexity. 6

  13. Design: Overview • Payoff and feedback mechanism: • All rules and mechanisms clearly communicated to subjects. • All points rewarded simultaneously after the experiment. • Goal: Collect as many points as possible. • Choosing x has 2 consequences: • Values initially unknown, but can be learned. • 6 abstract options (= colors): 7 • Subjects faced with sequence of 105 binary choices. , , , , , { } • Each color x associated with a payoff vector ( x 1 , x 2 ) x 1 + ϵ points shown and awarded immediately. x 2 + ϵ ′ points shown and awarded with one round delay. • ϵ , ϵ ′ are disturbances drawn uniformly from { 1 , 2 , 3 , 4 } . • Total value of x is x 1 + x 2

  14. Design: Overview • Payoff and feedback mechanism: • All rules and mechanisms clearly communicated to subjects. • All points rewarded simultaneously after the experiment. • Goal: Collect as many points as possible. • Choosing x has 2 consequences: • Values initially unknown, but can be learned. • 6 abstract options (= colors): 7 • Subjects faced with sequence of 105 binary choices. , , , , , { } • Each color x associated with a payoff vector ( x 1 , x 2 ) x 1 + ϵ points shown and awarded immediately. x 2 + ϵ ′ points shown and awarded with one round delay. • ϵ , ϵ ′ are disturbances drawn uniformly from { 1 , 2 , 3 , 4 } . • Total value of x is x 1 + x 2

  15. Design: Overview • Payoff and feedback mechanism: • All rules and mechanisms clearly communicated to subjects. • All points rewarded simultaneously after the experiment. • Goal: Collect as many points as possible. • Choosing x has 2 consequences: • Values initially unknown, but can be learned. • 6 abstract options (= colors): 7 • Subjects faced with sequence of 105 binary choices. , , , , , { } • Each color x associated with a payoff vector ( x 1 , x 2 ) x 1 + ϵ points shown and awarded immediately. x 2 + ϵ ′ points shown and awarded with one round delay. • ϵ , ϵ ′ are disturbances drawn uniformly from { 1 , 2 , 3 , 4 } . • Total value of x is x 1 + x 2

  16. Design: Example Screen 8

  17. Design: Example Screen 8

  18. Design: Example Screen 8

  19. Design: Example Screen 8

  20. Design: Example Screen 8

  21. Design: Example Screen 8

  22. Design: Example Screen 8

  23. Design: Example Screen 8

  24. Design: Example Screen 8

  25. Design: Example Screen 8

  26. Design: Example Screen 8

  27. Design: Example Screen 8

  28. Design: Example Screen 8

  29. Design: Example Screen 8

  30. Design: Example Screen 8

  31. Design: Example Screen 8

  32. Design: Example Screen 8

  33. Design: Payoff Vectors Group A Hypotheses: Option Group B 9 Payoff Vectors color e.g. ( total value ) ( immediate, delayed ) ( 18 ) ( 11 , 7 ) A ( 7 , 11 ) B ( 16 ) ( 6 , 10 ) A ( 10 , 6 ) B ( 14 ) ( 9 , 5 ) A ( 5 , 9 ) B ( 12 ) ( 4 , 8 ) A ( 8 , 4 ) B ( 10 ) ( 7 , 3 ) A ( 3 , 7 ) B ( 8 ) ( 2 , 6 ) A ( 6 , 2 ) B ( 11 , 7 ) A chosen more often than ( 7 , 11 ) B ; ( 10 , 6 ) B more than ( 6 , 10 ) A ; ... ( 11 , 7 ) A and ( 6 , 10 ) A further apart than ( 6 , 10 ) A and ( 9 , 5 ) A . Potentially even: ( 9 , 5 ) A preferred to ( 6 , 10 ) A .

  34. Design: Payoff Vectors Group A Hypotheses: Option Group B 9 Payoff Vectors color e.g. ( total value ) ( immediate, delayed ) ( 18 ) ( 11 , 7 ) A ( 7 , 11 ) B ( 16 ) ( 6 , 10 ) A ( 10 , 6 ) B ( 14 ) ( 9 , 5 ) A ( 5 , 9 ) B ( 12 ) ( 4 , 8 ) A ( 8 , 4 ) B ( 10 ) ( 7 , 3 ) A ( 3 , 7 ) B ( 8 ) ( 2 , 6 ) A ( 6 , 2 ) B ( 11 , 7 ) A chosen more often than ( 7 , 11 ) B ; ( 10 , 6 ) B more than ( 6 , 10 ) A ; ... ( 11 , 7 ) A and ( 6 , 10 ) A further apart than ( 6 , 10 ) A and ( 9 , 5 ) A . Potentially even: ( 9 , 5 ) A preferred to ( 6 , 10 ) A .

  35. Results: Choice Frequencies 10

  36. Results: Choice Frequencies 10

  37. Results: Choice Frequencies 10

  38. Results: Choice Frequencies 10

  39. Results: Choice Frequencies 10

  40. Results: Bias over time 11

  41. Summary: Further Results • Elicited beliefs are in accordance with choice behavior. • Considerable heterogeneity in degree of biasedness. • Correlated to impatience in hypothetical intertemporal choice. • (To do: Incentivized choice or field measures of impatience.) • Treatment: Learning by observation • Subjects passively presented with feedback for 63 rounds. • Directly afterwards: 42 own decisions. • Bias attenuated; low right after the learning phase, then gradually increasing. • Suggests emergence of bias is connected to active decision making. 12 • Estimated latent utility function: u ( x ) = x 1 + 0 . 4 x 2

  42. Summary: Further Results • Elicited beliefs are in accordance with choice behavior. • Considerable heterogeneity in degree of biasedness. • Correlated to impatience in hypothetical intertemporal choice. • (To do: Incentivized choice or field measures of impatience.) • Treatment: Learning by observation • Subjects passively presented with feedback for 63 rounds. • Directly afterwards: 42 own decisions. • Bias attenuated; low right after the learning phase, then gradually increasing. • Suggests emergence of bias is connected to active decision making. 12 • Estimated latent utility function: u ( x ) = x 1 + 0 . 4 x 2

Recommend

Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Lecture 11: Fast Reinforcement Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver Lecture 11: Fast Reinforcement Learning 3 Emma Brunskill (CS234 Reinforcement Learning. )

887 views • 66 slides
How Far Can Machines Go? Hila Peleg Technion Laziness Laziness Impatience Laziness

How Far Can Machines Go? Hila Peleg Technion Laziness Laziness Impatience Laziness

Automatic Programming: How Far Can Machines Go? Hila Peleg Technion Laziness Laziness Impatience Laziness Impatience Hubris Automatic Programming Clean up my spreadsheet! Right away, boss! Program synthesis to the rescue Program

1.01k views • 72 slides
Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2019 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

870 views • 65 slides
AONT-RS: Blending Security and Performance in Dispersed Storage Systems Jason Resch James Plank

AONT-RS: Blending Security and Performance in Dispersed Storage Systems Jason Resch James Plank

AONT-RS: Blending Security and Performance in Dispersed Storage Systems Jason Resch James Plank Cleversafe, Inc. University of Tennessee Chicago, IL Knoxville, TN 1 Topics Appeals of Dispersed Storage Methods for Securing Dispersed

659 views • 26 slides
Set Up Your Dev Setup {srinu, sayan} Three Chief Virtues of a Programmer: Laziness Impatience

Set Up Your Dev Setup {srinu, sayan} Three Chief Virtues of a Programmer: Laziness Impatience

Set Up Your Dev Setup {srinu, sayan} Three Chief Virtues of a Programmer: Laziness Impatience Hubris Three Focus Areas for Today: Package Management Writing and Executing Code Common Frameworks This is very subjective!

674 views • 23 slides
Oregon Behavioral Health Integration: What weve heard, seen and learned Integration

Oregon Behavioral Health Integration: What weve heard, seen and learned Integration

Oregon Behavioral Health Integration: What weve heard, seen and learned Integration Environmental Scan OHSU qualitative research, five geographically dispersed CCO communities. Additional 30+ site visits, CCO and provider interviews.

207 views • 18 slides
Experimental choices and field behavior on impatience, saving and smoking Matthias Sutter

Experimental choices and field behavior on impatience, saving and smoking Matthias Sutter

Latsis Symposium 2012 Economics on the Move Experimental choices and field behavior on impatience, saving and smoking Matthias Sutter University of Innsbruck and University of Gothenburg 1 Overview of talk Main paper (forthcoming

554 views • 34 slides
Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Lecture 12: Fast Reinforcement Learning Part II 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver, Worked Examples New Lecture 12: Fast Reinforcement Learning Part II 3 Emma Brunskill

1.04k views • 70 slides
Reinforcement Learning II George Konidaris gdk@cs.brown.edu Fall 2019 Reinforcement Learning

Reinforcement Learning II George Konidaris gdk@cs.brown.edu Fall 2019 Reinforcement Learning

Reinforcement Learning II George Konidaris gdk@cs.brown.edu Fall 2019 Reinforcement Learning t r t R = max : S A t =0 MDPs Agent interacts with an environment At each time t: Receives sensor signal s t

717 views • 40 slides
THERMOELECTRIC PROPERTIES OF Cu-DISPERSED Bi 2 Te 2.7 Se 0.3 NANOCOMPOSITES Il-Ho Kim 1 *, Soon-Mok

THERMOELECTRIC PROPERTIES OF Cu-DISPERSED Bi 2 Te 2.7 Se 0.3 NANOCOMPOSITES Il-Ho Kim 1 *, Soon-Mok

18 TH INTERNATIONAL CONFERENCE ON COMPOSITE MATERIALS THERMOELECTRIC PROPERTIES OF Cu-DISPERSED Bi 2 Te 2.7 Se 0.3 NANOCOMPOSITES Il-Ho Kim 1 *, Soon-Mok Choi 2 , Won-Seon Seo 2 , Dong-Ik Cheong 3 , Hyung Kang 3 1 Department of Materials Science

100 views • 5 slides
Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David

Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David

Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David Silver, Dan Klein, Mausam, Dan Weld Reinforcement Learning At each time step t : Agent executes an action A t Environment emits a reward Rt

1.1k views • 71 slides
Reinforcement Learning, cont. Models an agent trying to learn what to do / how to act

Reinforcement Learning, cont. Models an agent trying to learn what to do / how to act

Reinforcement Learning, cont. Models an agent trying to learn what to do / how to act in a complex environment. So far, we have seen passive learning We learned utility values for a fixed world with fixed transitions (no real

737 views • 29 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Class Structure Last time: Batch RL This Time: MCTS Next time: Human in the Loop RL Lecture 16:

Class Structure Last time: Batch RL This Time: MCTS Next time: Human in the Loop RL Lecture 16:

Lecture 16: MCTS 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 1 With many slides from or derived from David Silver Lecture 16: MCTS 1 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 / 57 Class Structure Last time:

1.28k views • 58 slides
Class Structure Last time: Batch RL This Time: MCTS Next time: Human in the Loop RL Lecture 14:

Class Structure Last time: Batch RL This Time: MCTS Next time: Human in the Loop RL Lecture 14:

Lecture 14: MCTS 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver Lecture 14: MCTS 3 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 / 57 Class Structure Last time:

817 views • 57 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall

DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall

This lecture will be recorded!!! Welcome to DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall 2020 Last Lecture v What is reinforcement learning? v Difference from other AI problems v Application

1.04k views • 52 slides
DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall

DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall

This lecture will be recorded!!! Welcome to DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall 2020 Last lecture v Reinforcement Learning Components Model, Value function, Policy v Model-based

944 views • 78 slides
Board Ac)on on the Report Inges)on and Effects of Dispersed Oil on Marine Zooplankton by

Board Ac)on on the Report Inges)on and Effects of Dispersed Oil on Marine Zooplankton by

5/1/13 Board Ac)on on the Report Inges)on and Effects of Dispersed Oil on Marine Zooplankton by Richard Lee Skidaway Ins7tute of Oceanography, Savannah, GA C130 Hercules Drops Dispersant on the Deepwater Horizon Spill 1 5/1/13 Agenda

294 views • 7 slides
CULE*: GPU ACCELERATED RL * CUDA Learning Environment Reinforcement Learning Steven Dalton, Iuri

CULE*: GPU ACCELERATED RL * CUDA Learning Environment Reinforcement Learning Steven Dalton, Iuri

March 26-29, 2018 | Silicon Valley CULE*: GPU ACCELERATED RL * CUDA Learning Environment Reinforcement Learning Steven Dalton, Iuri Frosio, Jared Hoberock, Jason Clemons time REINFORCEMENT LEARNING A successful approach Board games

551 views • 41 slides
4303 Ed/Serious Feedback and Reinforcement Feedback and Reinforcement Both of them are

4303 Ed/Serious Feedback and Reinforcement Feedback and Reinforcement Both of them are

4303 Ed/Serious Feedback and Reinforcement Feedback and Reinforcement Both of them are the most important concepts in learning. Any differences? Feedback Feedback involves providing learners with information about their

710 views • 24 slides
KEY STAR TECHNOLOGIES: DISPERSED MULTIPHASE FLOW AND LIQUID FILM MODELLING DAVID GOSMAN EXEC VP

KEY STAR TECHNOLOGIES: DISPERSED MULTIPHASE FLOW AND LIQUID FILM MODELLING DAVID GOSMAN EXEC VP

KEY STAR TECHNOLOGIES: DISPERSED MULTIPHASE FLOW AND LIQUID FILM MODELLING DAVID GOSMAN EXEC VP TECHNOLOGY, CD-adapco INTRODUCTION KEY METHODOLOGIES AVAILABLE IN STAR-CCM+ AND STAR-CD 1. Lagrangian modelling of dispersed multiphase flow of

681 views • 27 slides
7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views • 56 slides
What we learned last time 1. Intelligence is the computational part of the ability to achieve

What we learned last time 1. Intelligence is the computational part of the ability to achieve

What we learned last time 1. Intelligence is the computational part of the ability to achieve goals looking deeper: 1) its a continuum, 2) its an appearance, 3) it varies with observer and purpose 2. We will (probably) figure out how to make

455 views • 32 slides

More recommend