Eliciting Informative Feedback: The Peer-Prediction Method Colin - PowerPoint PPT Presentation

Eliciting Informative Feedback: The Peer-Prediction Method Colin Zheng and Kenneth Wang 1

The Setting Camera Quality: dist 𝐸 over {𝑀, 𝐼} 𝑡 2 ~𝑇 𝑡 1 ~𝑇 𝑇 : dist over 𝑚, ℎ jointly Rater Rater w/ 𝐸 1 2 Report Report 𝑠 1 𝑠 2 Center: - Ebay - NetFlix - Amazon 2 2

The Setting Camera Quality: dist 𝐸 over {𝑀, 𝐼} Task: 𝑡 2 ~𝑇 𝑡 1 ~𝑇 𝑇 : dist over How to make rational 𝑚, ℎ jointly raters report honestly Rater Rater w/ 𝐸 ( 𝑠 𝑗 = 𝑡 𝑗 ) 1 2 Naive Attempt: reward 𝜐 = $1 if 𝑠 1 = 𝑠 2 , 𝜐 = $0 otherwise Report Report Problem: rater 1 will report 𝑠 1 𝑠 2 “more likely signal” of rater 2 Center: - Ebay - NetFlix - Amazon 3 3

The Setting Camera Quality: dist 𝐸 over {𝑀, 𝐼} Task (formulation): Reporting 𝑠 𝑗 = 𝑡 𝑗 is a 𝑡 2 ~𝑇 𝑡 1 ~𝑇 𝑇 : dist over Nash Equilibrium 𝑚, ℎ jointly Rater Rater w/ 𝐸 1 2 ∀ rater 𝑗 , signal 𝑛 , 𝑦 ≠ 𝑛 𝔽 𝑡 2 𝜐 1 𝑡 1 , 𝑡 2 𝑡 1 = 𝑛 ≥ 𝔽 𝑡 2 𝜐 1 𝑦, 𝑡 2 𝑡 1 = 𝑛 Report Report 𝑠 1 𝑠 2 𝔽 𝑡 1 𝜐 2 𝑡 2 , 𝑡 1 𝑡 2 = 𝑛 Center: ≥ 𝔽 𝑡 1 𝜐 2 𝑦, 𝑡 1 𝑡 2 = 𝑛 - Ebay - NetFlix - Amazon 4 4

Solution using Proper Scoring Rule Camera Quality: Task: dist 𝐸 over {𝑀, 𝐼} choose payoff 𝜐 1 such that 𝔽 𝑡 2 𝜐 1 𝑦, 𝑡 2 𝑡 1 = 𝑛 𝑡 2 ~𝑇 𝑡 1 ~𝑇 is maximized at 𝑦 = 𝑛 𝑇 : dist over 𝑚, ℎ jointly Rater Rater w/ 𝐸 1 2 Equivalently, 𝔽 𝑨~𝑎 𝜐 1 𝑦, 𝑨 where 𝑎 = 𝑡 2 𝑡 1 =𝑛 Report Report 𝑠 1 𝑠 2 Recall: 𝑔 is Proper Scoring Rule if 𝔽 𝑨~𝑎 [𝑔(𝑄, 𝑨)] is maximized at Center: 𝑄 = 𝑎 - Ebay - NetFlix So let 𝜐 1 𝑦, 𝑨 = 𝑔(𝑡 2 𝑡 1 =𝑦 , 𝑨) - Amazon 5 5

Issue: costs to the rater In reality,  Evaluating and reporting honestly incur a relative cost 𝑑 > 0 (e.g. testing is time consuming, opportunity cost, ...)  Can be offset by scaling up the payoffs  Problems: – Paying too much for truthful information? – What if 𝑑 is unknown? [PRGJ08] 6 6

Issue: costs to the center To reduce center’s cost:  Budget balancing: – pair up raters, make each pair a zero-sum game  Use linear optimization to find cost-optimal payoff function [JF06] 7 7

Issue: risk aversion  When the center knows the raters’ utility function(s)  When the center does not know the utility function(s)  Using multiple reference raters reduces risk 8 8

Issue: collusion  Using an honest reference rater  Randomized reference rater  Outside experts... 9 9

Issue: Unknown/diff. priors Camera Quality: dist 𝐸 over {𝑀, 𝐼}  In reality, raters have diff. beliefs about 𝑡 2 ~𝑇 𝑡 1 ~𝑇 𝑇 : dist over (𝐸, 𝑇) , unknown to 𝑚, ℎ jointly Rater Rater w/ 𝐸 the center 1 2  Addressed in [WP12] for binary signals 𝑇 Report Report 𝑠 1 𝑠 2 Center: - Ebay - NetFlix - Amazon 10 10