Bayesian Post-Election Audits Ronald L. Rivest and Emily Shen - PowerPoint PPT Presentation

Bayesian Post-Election Audits Ronald L. Rivest and Emily Shen Viterbi Professor of EECS MIT, Cambridge, MA {rivest,ehshen}@mit.edu EVT/WOTE 2012 2012-08-07 1

Outline Post-Election Audits Bayesian Ballot-Polling Bayesian Comparison Audits Experimental Results Lessons and Open Questions 2

Post-Election Audit Objectives By examining by hand sufficiently many randomly selected paper ballots: ◮ Confirm to a high degree of confidence that the reported (scanner-based) outcome is correct or else that the actual (full hand-count) outcome is different. 3

Post-Election Audit Objectives By examining by hand sufficiently many randomly selected paper ballots: ◮ Confirm to a high degree of confidence that the reported (scanner-based) outcome is correct or else that the actual (full hand-count) outcome is different. ◮ Convince the losers they really lost! 4

Single-ballot Audits ◮ Sequential decision-making (Wald). ◮ Examine paper ballots one at a time , in random order. ◮ Determine actual type of each ballot (as opposed to its reported type ). ◮ At each stage, decide whether to ◮ Stop: Reported outcome looks OK . ◮ Continue: more auditing needed. (We assume that full hand count needed to overturn reported outcome.) 5

Ballot-polling audits vs. comparison audits ◮ Ballot-polling audit: look at only the actual ballot types of the audited ballots: 6

Ballot-polling audits vs. comparison audits ◮ Ballot-polling audit: look at only the actual ballot types of the audited ballots: actual types (by hand): 7

Ballot-polling audits vs. comparison audits ◮ Ballot-polling audit: look at only the actual ballot types of the audited ballots: actual types (by hand): ? ? ? ? ? 8

Ballot-polling audits vs. comparison audits ◮ Ballot-polling audit: look at only the actual ballot types of the audited ballots: actual types (by hand): A ? ? ? ? 9

Ballot-polling audits vs. comparison audits ◮ Ballot-polling audit: look at only the actual ballot types of the audited ballots: actual types (by hand): A A ? ? ? 10

Ballot-polling audits vs. comparison audits ◮ Ballot-polling audit: look at only the actual ballot types of the audited ballots: actual types (by hand): A A ? ? ? ◮ Comparison audit: also look at their reported types: 11

Ballot-polling audits vs. comparison audits ◮ Ballot-polling audit: look at only the actual ballot types of the audited ballots: actual types (by hand): A A ? ? ? ◮ Comparison audit: also look at their reported types: reported types (scanner): A A A B A 12

Ballot-polling audits vs. comparison audits ◮ Ballot-polling audit: look at only the actual ballot types of the audited ballots: actual types (by hand): A A ? ? ? ◮ Comparison audit: also look at their reported types: reported types (scanner): A A A B A actual types (by hand): ? ? ? ? ? 13

Ballot-polling audits vs. comparison audits ◮ Ballot-polling audit: look at only the actual ballot types of the audited ballots: actual types (by hand): A A ? ? ? ◮ Comparison audit: also look at their reported types: reported types (scanner): A A A B A actual types (by hand): A ? ? ? ? 14

Ballot-polling audits vs. comparison audits ◮ Ballot-polling audit: look at only the actual ballot types of the audited ballots: actual types (by hand): A A ? ? ? ◮ Comparison audit: also look at their reported types: reported types (scanner): A A A B A actual types (by hand): A A ? ? ? 15

Auditing with a magic box ◮ Assume you audit randomly chosen ballots, one by one, in a ballot-polling audit. 16

Auditing with a magic box ◮ Assume you audit randomly chosen ballots, one by one, in a ballot-polling audit. ◮ Suppose I give you a “magic box” that at any time can answer the question, 17

Auditing with a magic box ◮ Assume you audit randomly chosen ballots, one by one, in a ballot-polling audit. ◮ Suppose I give you a “magic box” that at any time can answer the question, Given what you’ve seen in the audit so far, what is the probability that each candidate would win if all ballots were examined? 18

Auditing with a magic box ◮ Assume you audit randomly chosen ballots, one by one, in a ballot-polling audit. ◮ Suppose I give you a “magic box” that at any time can answer the question, Given what you’ve seen in the audit so far, what is the probability that each candidate would win if all ballots were examined? ◮ Then you can stop audit if/when the reported winner has at least (say) 95% probability of winning. 19

An example Actual ballot types (by hand): ? ? ? ? ? ? ? ? ? ? . . . ProbabilityAwins: 50.0% ProbabilityBwins: 50.0% 20

An example Actual ballot types (by hand): A ? ? ? ? ? ? ? ? ? . . . ProbabilityAwins: 75.0% ProbabilityBwins: 25.0% 21

An example Actual ballot types (by hand): A A ? ? ? ? ? ? ? ? . . . ProbabilityAwins: 87.5% ProbabilityBwins: 12.5% 22

An example Actual ballot types (by hand): A A B ? ? ? ? ? ? ? . . . ProbabilityAwins: 68.8% ProbabilityBwins: 31.2% 23

An example Actual ballot types (by hand): A A B B ? ? ? ? ? ? . . . ProbabilityAwins: 50.0% ProbabilityBwins: 50.0% 24

An example Actual ballot types (by hand): A A B B A ? ? ? ? ? . . . ProbabilityAwins: 65.6% ProbabilityBwins: 34.4% 25

An example Actual ballot types (by hand): A A B B A A ? ? ? ? . . . ProbabilityAwins: 77.4% ProbabilityBwins: 22.6% 26

An example Actual ballot types (by hand): A A B B A A A ? ? ? . . . ProbabilityAwins: 85.6% ProbabilityBwins: 14.4% 27

An example Actual ballot types (by hand): A A B B A A A A ? ? . . . ProbabilityAwins: 91.0% ProbabilityBwins: 9.0% 28

An example Actual ballot types (by hand): A A B B A A A A A ? . . . ProbabilityAwins: 94.5% ProbabilityBwins: 5.5% 29

An example Actual ballot types (by hand): A A B B A A A A A A. . . ProbabilityAwins: 96.7% ProbabilityBwins: 3.3% 30

An example Actual ballot types (by hand): A A B B A A A A A A. . . ProbabilityAwins: 96.7% ProbabilityBwins: 3.3% → Stop auditing! ← 31

Making the magic box (ballot-polling) ◮ Suppose you are auditing an election between candidates A and B, with 5 ballots. 32

Making the magic box (ballot-polling) ◮ Suppose you are auditing an election between candidates A and B, with 5 ballots. ◮ You draw a random sample (without replacement) of two ballots. 33

Making the magic box (ballot-polling) ◮ Suppose you are auditing an election between candidates A and B, with 5 ballots. ◮ You draw a random sample (without replacement) of two ballots. ◮ Both ballots are for A: A ? A ? ? 34

Making the magic box (ballot-polling) ◮ Suppose you are auditing an election between candidates A and B, with 5 ballots. ◮ You draw a random sample (without replacement) of two ballots. ◮ Both ballots are for A: A ? A ? ? ◮ Q: What is the probability that A won? 35

Answer (Bayesian) ◮ To make Q well-posed, need a model (a prior ) for the likelihood of different outcomes. 36

Answer (Bayesian) ◮ To make Q well-posed, need a model (a prior ) for the likelihood of different outcomes. ◮ A noninformative prior gives each outcome (A:B tally) equal probability: tally 5:0 4:1 3:2 2:3 1:4 0:5 Prob 1/6 1/6 1/6 1/6 1/6 1/6 37

Answer (Bayesian) ◮ To make Q well-posed, need a model (a prior ) for the likelihood of different outcomes. ◮ A noninformative prior gives each outcome (A:B tally) equal probability: tally 5:0 4:1 3:2 2:3 1:4 0:5 Prob 1/6 1/6 1/6 1/6 1/6 1/6 ◮ With this prior and sample, A wins with (subjective) probability 95 % 38

Answer (Bayesian) ◮ To make Q well-posed, need a model (a prior ) for the likelihood of different outcomes. ◮ A noninformative prior gives each outcome (A:B tally) equal probability: tally 5:0 4:1 3:2 2:3 1:4 0:5 Prob 1/6 1/6 1/6 1/6 1/6 1/6 ◮ With this prior and sample, A wins with (subjective) probability 95 % ◮ If your error limit is 5%, stop auditing! 39

95%? (Bayes Rule) posterior probability proportional to: prior times likelihood of sample given prior 40

95%? (Bayes Rule) posterior probability proportional to: prior times likelihood of sample given prior tally 5:0 4:1 3:2 2:3 1:4 0:5 prior 1/6 1/6 1/6 1/6 1/6 1/6 41

95%? (Bayes Rule) posterior probability proportional to: prior times likelihood of sample given prior tally 5:0 4:1 3:2 2:3 1:4 0:5 prior 1/6 1/6 1/6 1/6 1/6 1/6 5 5 · 4 4 5 · 3 3 5 · 2 5 · 1 2 likelihood(A A) 0 0 4 4 4 4 42

95%? (Bayes Rule) posterior probability proportional to: prior times likelihood of sample given prior tally 5:0 4:1 3:2 2:3 1:4 0:5 prior 1/6 1/6 1/6 1/6 1/6 1/6 5 · 4 5 5 · 3 4 3 5 · 2 2 5 · 1 likelihood(A A) 0 0 4 4 4 4 10 6 3 1 product 0 0 60 60 60 60 43

95%? (Bayes Rule) posterior probability proportional to: prior times likelihood of sample given prior tally 5:0 4:1 3:2 2:3 1:4 0:5 prior 1/6 1/6 1/6 1/6 1/6 1/6 5 5 · 4 4 5 · 3 3 5 · 2 5 · 1 2 likelihood(A A) 0 0 4 4 4 4 10 6 3 1 product 0 0 60 60 60 60 10 6 3 1 posterior 0 0 20 20 20 20 44