Emergent systems Spring-13 The Prisoners Dilemma - PDF document

Umeå University Department of Computing Science Emergent systems Spring-13 The Prisoners ´ Dilemma http://www.cs.umu.se/kurser/5DV017 Last time ❒ Evolutionary computation ❍ Overview ❒ Genetic programming ❒ Aspects of evolution ❒ Classifier systems ❒ General on cooperation ❒ Game theory 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU Today ❒ Prisoners’ Dilemma and other dilemmas ❍ Iterated Prisoners’ Dilemma ❍ Ecological models ❍ Spatial models ❒ Short conclusion of the course 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU 1

The Prisoners’ Dilemma ❒ Melvin Dresher and Merrill Flood, RAND Corporation, 1950 ❒ Further developed by Albert W. Tucker ❒ Used in philosophy, ethics, biology, sociology, political science, economics, game theory, computer science, mathematics, ... ❒ ” The E. Coli of social psychology ” - Axelrod 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU The Prisoners’ Dilemma ❒ The story ❍ Two criminals were caught, they have committed a crime but the police have insufficient evidence ❍ They can not communicate with each other ❍ If both plead guilty, each get 10 years ❍ If one plead guilty and accuses the other • The one that plead guilty goes free • The accused get 20 years ❍ If no one plead guilty, each get 1 year 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU The Prisoners’ Dilemma ❒ Payoff matrix Bob Cooperate Defect Cooperate -1, -1 -20, 0 Alice Defect 0, -20 -10, -10 ❒ Alice rational analysis (dominant strategy) ❍ If Bob cooperate – Defect ❍ If Bob defect – Defect ❍ (Same analysis for Bob) ❒ Dominant strategy equilibrium � 10 year each ❒ Make an irrational decision (coop) � 1 year each 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU 2

The Prisoners’ Dilemma ❒ Summary ❍ Individual rationality is not optimal ❍ An example of a dilemma in game theory 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU Dilemmas in game theory ❒ ”A situation requiring a choice between alternatives that are equivalent” ❒ ” Damned if you do, damned if you don’t ” ❒ In game theory: Each player acts rationally, but the result is not desirable 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU Dilemmas in game theory ❒ General payoff matrix Bob Cooperate Defect CC (R) CD (S) Cooperate Reward Sucker’s payoff Alice DC (T) DD (P) Defect Temptation to Punishment defect 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU 3

Bob Dilemmas Coop Def Coop CC CD Alice Def DC DD ❒ General terms of a dilemma ❍ You will always win if the other cooperates • CC > CD and DC > DD ❍ Sometimes you win by defecting • DC > CC or DD > CD ❍ Mutual cooperation is preferable • CC > DD ❒ 24 permutations but only 3 are dilemmas 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU Bob Dilemmas Coop Def Coop CC CD Alice Def DC DD ❒ Prisoners’ Dilemma ❍ DC > CC > DD > CD ❍ Better to defect no matter what the other do ❍ Nash equilibrium is DD ❒ Chicken ❍ DC > CC > CD > DD ❍ Mutual defect is worst ❍ Two Nash equilibrium, DC and CD ❒ Stag Hunt ❍ CC > DC > DD > CD ❍ Best to cooperate with cooperators ❍ Two Nash equilibrium, CC and DD 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU Iterated Prisoners’ Dilemma ❒ Assumptions ❍ No agreements or threats ❍ A player's next move can not be predicted ❍ No way to eliminate players or avoid interaction ❍ No way of changing the payoff ❍ Communication only via direct interaction 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU 4

Iterated Prisoners’ Dilemma ❒ Axelrod’s experiment (1980) ❍ Intuitive assumption that future interactions may affect the rationality of the decision ❍ Round-robin tournament for strategies • All are competing against everyone, including itself • 200 iterations of PD – Should be unknown! • Each program/strategy can remember earlier actions • 14 program were received • CC = 3, CD = 0, DC = 5, DD = 1 Bob • (Requirement : DC + CD < 2 * CC) Coop Def Coop 3, 3 0, 5 Alice Def 5, 0 1, 1 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU Iterated Prisoners’ Dilemma ❒ Expected payoff for three simple strategies ALL-C RAND ALL-D Medel ALL-C 3.0 1.5 0.0 1.5 RAND 4.0 2.0 0.5 2.167 ALL-D 5.0 3.0 1.0 3.0 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU Iterated Prisoners’ Dilemma ❒ Result ❍ Winner: Rapoport’s Tit-For-Tat (TFT) ❍ Cooperate at first interaction ❍ Then, do what your opponent did the previous move ❒ Second experiment ❍ 62 program ❍ Everyone knew that TFT won last time ❍ TFT won again ❒ Tit-for-Two-Tats ❍ More forgiving than TFT ❍ TFT is better in a noisy environment ❍ Would have won the first experiment, but performed poorly in the second 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU 5

Iterated Prisoners’ Dilemma ❒ Characteristics for a successful strategy ❍ Don’t be envious ❍ Be nice ❍ Reciprocate ❍ Don’t be too clever ❍ Be a generalist ❍ Agree with yourself ❍ Be an evolutionarily stable strategy • John Maynard Smith • Resistant against invasion by other strategies 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU Ecological models ❒ What happens if we allow more successful strategies to spread in the population at the expense of less successful? ❒ Success is measured as a percentage of population ❒ Percentage of population = The probability that a random program uses this strategy ❒ On the board ❒ Simulation ❍ Fig 17.3a 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU Pavlov – new strategy ❒ Pavlov (PAV) ❍ ”Win-Stay, Lose-Shift” ❍ Start with cooperation ❍ If the other cooperate, continue with the current behavior ❍ If the other defect, switch behavior ❍ Fig 17.3b 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU 6

Noise ❒ An action is misinterpreted for some reason ❒ TFT ❍ Leads to a cycle of alternating CD and DC ❍ Broken only by a further error ❒ PAV ❍ Can self-correct itself : DC � DD � CC ❍ Can exploit ALL-C in an environment where errors can occur ❍ Fig 17.3c 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU IPD and GA ❒ Let the population evolve using genetic algorithms ❒ Simulation facts : ❍ N = 100 ❍ 50 bouts/individual/generation with each bout consisting of 20 PD rounds ❍ Random strategies initially ❍ Coding: Strings with five letters, each C or D First CC CD DC DD 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU IPD and GA (cont.) ❒ Result 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU 7

Stimulating cooperation in PD ❒ Know something about the other ❍ Iterative ❍ Rumor ❒ Spatial ❍ Interact with neighbors ❍ Example • MANET • Society 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU 2 x 2 game ❒ 2 players, 2 strategies ❒ Payoff matrix Coop Def Coop R = 1 S Def T P = 0 ❒ 12 permutations of R, S, T, P ❍ Fig 1 in the article by Hauert 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU 2 x 2 game Well mixed populations ❒ Fig 2 in the article by Hauert ❒ Defection: S < 0, T > 1 a) • Stable fix point = 0 (proportion of cooperators) • The area for Prisoners’ Dilemma b) Co-existence: S > 0, T > 1 • Stable fix point = S / (S + T – 1) Bi-stability: S < 0, T < 1 c) • Instable fix point = S / (S + T – 1) • Stable fix points = 0, 1 d) Cooperation: S > 0, T < 1 • Stable fix point = 1 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU 8

Spatial Prisoners’ Dilemma ❒ The overall payoff determines an individual's success and is determined by interaction with neighbors ❒ Each individual has a place in a grid ❒ In each generation N individuals are selected ❒ N = number of locations in the grid ❒ Every individual play against its neighbors and then each individual get the opportunity to update its strategy 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU Spatial PD - Variations ❒ Updating of the grid ❍ Synchronous ❍ Asynchronous ❍ Delayed asynchronous ❒ Updating of individuals ❍ Best takes over (deterministic) ❍ Proportional updating ❒ Number of neighbors ❍ von Neumann – 4 ❍ Moore – 8 ❒ Initial frequency of cooperators ❍ 0.2, 0.5 or 0.8 ❒ Simulation 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU Spatial PD - Discussion ❒ Hauert (2001) ❍ Spatial extensions has an effect ❍ Differences in initial frequency of cooperators will not affect ❍ Increased stochasticity in updating the grid has surprisingly little impact ❍ Increased stochasticity in updating of individuals decrease cooperation 21/2 - 13 Emergent Systems, Jonny Pettersson, UmU 9