Scoring rules A different kind of mechanism design problem: how to - PowerPoint PPT Presentation

Scoring rules • A different kind of mechanism design problem: how to elicit a good prediction of an uncertain event? – Weather forecaster: will it rain tomorrow? – Political pundit: will a Democrat or Republican win next election – Microsoft employee: will the next version of MS Office ship on time? • How should we evaluate the quality of a prediction/pay based on the quality of predictions/ incentivize the work needed to output the best possible prediction?

Scoring rules • X finite set of possible outcomes of uncertain event. rain snow X sun • A scoring rule is a real-valued function S(q,i) – q is a probability distribution over X (a prediction) – i is some outcome in X (the realized outcome) outcome nuns 9 s doesnt I 49,94 if

x spy's sy To Model for incentives FpFpatpz l i EX • Forecaster has a belief p, prob distribution over X. • Forecaster will choose prediction q to maximize expected score forecaster's goal FE miaF p report of sepsis

Strictly proper scoring rules • X finite set of possible outcomes of uncertain event. • A scoring rule is a real-valued function S(q,i) – q is a probability distribution over X (a prediction) – i is some outcome in X (the realized outcome) • A scoring rule is strictly proper if, no matter what the true belief p of the forecaster is, her unique best response is to report truthfully, i.e. to set q = p.

Strictly proper scoring rules • X finite set of possible outcomes of uncertain event. • A scoring rule is a real-valued function S(q,i) – q is a probability distribution over X (a prediction) – i is some outcome in X (the realized outcome) piscopi A scoring rule is strictly proper if, no matter what the true belief p of the • forecaster is, her unique best response is to report truthfully, i.e. to set q = p. S pi bebfCp g Example I g g report Exp payoff p 9 th tg p oEo3Ct this maximes gwent what g g T p

Quadratic scoring rule IS.FI sfoiit ai 12 12 if it happens I for some 9 1 qi V J 4 t 0 doesnt 95 if i 2 happen 3 LT no matter wathatpaydt 9i th

9F Sff if gi QSRissmctypropesees.mx Pi t2 afffff x9 Pioli z Pi 9k Elson d pie P a z

Logarithmic scoring rule ln S qi 1 1 4 en 1 1 add nonreg exp utility guarantee forecaster can h Ff't'm O then 2 pinned F sore

is strictly rule Logarithmic scoring proper honest feedback incenhrizey markets prediction

Incentivizing honest feedback • Example: peer grading, where students grade the assignments of other students. • How to incentivize accurate grading, without direct verification?

Model • n players (graders of an assignment, say in MOOC) EYd gm • Player i has a “signal” ! " Dff • Each player submits a report # " to a mechanism. • Mechanism pays player $ " (# & , … , # ) ) Assume signalsfsinsn drawntramcorrelateddistin E aM SEED sbsido E H bad 89 E 1 good 6

E Incry payment As IT How to choose truffle to incentivize reporting

Output Agreement reward agreement • For each player ! – Pick a random player " ≠ ! – Set payoff $ % equal to 1 if they agree, 0 otherwise.

D

Output Agreement • For each player ! – Pick a random player " ≠ ! – Set payoff $ % equal to 1 if they agree, 0 otherwise. a Nash eg to report tmhfully Is it s Prfs2 x sp oyez 7 stood prlsj yfsi xlsmsi bad o s t Tt V y iio.io everyone report good bad NE has Mechanism

Peer prediction mechanism Suppose the distribution ! over signals is known to mechanism. • • For each player " – Pick a random player # ≠ " – Let ! % (' ( ) be the distribution of * % conditioned on * ( = ' ( 9T i so – Set " ’s payoff , ( ≔ . ! % ' ( , ' % players repaid prediction of the distribution treat a of other player's signal Dfo In 07 24 O in o tM bad sq.t.TT SEE good

Problems • Requires advance knowledge of distribution. • Other non-truthful and “bad” equilibria. • In experiments: – Participants coordinate on high-payoff but Ed uninformative equilibria – Empirically, people give better/truthful reports when paid a fixed reward (indep of their report).

Prediction Markets • Suppose you’re interested in an uncertain event e.g., – Will Trump be reelected? – Will there be a Covid-19 vaccine by the end of 2020? – Who will win the next superbowl? market for uncertain events Pred market stock like political onteenes Predict It IEM

Prediction markets • Idea: say want to predict which of two candidates A or B will win election. • Create two securities a and b: – Each share of security a will pay out $1 if A wins. – Each share of security b will pay out $1 if B wins. • Allow people to buy and sell these securities. • Suppose current price of a is 75 cents (and b is 25 cents) and you believe A will win with probability p. • What do you do?

O I

you believe that chance will win is 5 that Trump 0.49 10.52 Exp payoff 0 03

Prediction markets • Idea: say want to predict which of two candidates A or B will win election. • Create two securities a and b: – Each share of security a will pay out $1 if A wins. – Each share of security b will pay out $1 if B wins. • Allow people to buy and sell these securities. • Interpret market price as the market’s “belief” that the candidate will win the election. • Market aggregating beliefs of all participants => “consensus opinion”.

Legality Issues • IEM, PredictIt circumvent regulation through a no-action letter by CFTC which condones IEM – Non-profit and used for research purposes – Stakes are small • Several prediction markets with fictitious currency. • No real path to establishing legal real-money prediction markets.

Accuracy • Prediction markets vs polls • Historically, prediction markets have done pretty well – People are better at predicting what other people will do than themselves. Bad in 2016

Basic prediction market (e.g. IEM) • Use continuous double auctions – Trader can submit a buy or sell order any time. – An order: • Price • Max number of shares to be bought/sold. • Expiration date. – Trades are executed greedily (with nuances).

w y I B h m ixia i 3.58k

The Wisdom of Crowds [Surowiecki] (2004) HP in 90 s ran goobles Google trust

Another Approach – Market Scoring Rules • CDAs work well for “thick” markets – lots of traders, but not in – “thin” markets – few traders – “illiquid” markets -- large “bid-ask spread” • Different approach: automated market-maker – At any time there is a price, and the market is always happy to buy or sell shares at this price. – Price evolves as shares are bought and sold.

Automated Market Makers • Implemented using strictly proper scoring rule that is “shared” by all the players. • Let S be a strictly proper scoring rule. over X th th tn dish Initialize p atanghet pt can update p any player is realized P iEX outcome when to players who pH pt update payout s is scp

to extent to which ont pad according improved predechn report Properties specifically Market maher has bounded financial loss grog song rule far T steps if a rung S pii S pti totalpagat th ve honey byte SCp9i A

Ifand PYEspafg.msEradesonceinafxedordeDJ best response for to them eachplayer unique to their true belief update my true belief p to max report pt Iwle Sept D pti Ei a report pt in best interest to p.es Ewu3oErmeqpFiIQ Alice Suppose knows ontane of coin'd 4,24 tails that its knows she should report 0,1 s es vB Its 2nd coin toss atone was g if Bob up oil

What does this do? • Player is rewarded according to extent her report improves the prediction. • Final prediction is last distribution. • Predictions tend to settle down.