Consensus Agreement Games Paul Upchurch, Daniel Sedra, Andrew - PowerPoint PPT Presentation

Consensus Agreement Games Paul Upchurch, Daniel Sedra, Andrew Mullen, Haym Hirsh and Kavita Bala Cornell University HCOMP 2016

Lamp Plate Plate

Glass Food Fabric

Dining room Talking Eating

OpenSurfaces, hard-to-label examples Ceramic or glass?

OpenSurfaces, hard-to-label examples Ceramic or Painted or glass? wood?

OpenSurfaces, hard-to-label examples Ceramic or Painted or Ceramic, plastic glass? wood? or foliage?

OpenSurfaces, hard-to-label examples Ceramic or Painted or Ceramic, plastic Plastic or glass? wood? or foliage? glass?

Worker Error Rate by Agreement paint paint paint paint paint paint paint paint 0% error MTurk workers Experts

Worker Error Rate by Agreement paint paint paint paint paint paint paint paint 0% error paint paint paint paint paint paint paint wood 5% error MTurk workers Experts

Worker Error Rate by Agreement paint paint paint paint paint paint paint paint 0% error paint paint paint paint paint paint paint wood 5% error paint paint paint paint paint paint wood wood 27% error MTurk workers Experts

Worker Error Rate by Agreement paint paint paint paint paint paint paint paint 0% error paint paint paint paint paint paint paint wood 5% error paint paint paint paint paint paint wood wood 27% error MTurk workers Experts

Get More Worker Opinions? 7 votes 19.4% error

Get More Worker Opinions? 7 votes 19.4% error 11 votes 19.0% error

Get More Worker Opinions? 7 votes 19.4% error 11 votes 19.0% error 21 votes 19.6% error

Cost vs. Quality on 3 out of 5 images MTurk 7 votes Error rate Expert Labels Cost

Cost vs. Quality MTurk MTurk MTurk 7 votes 11 votes 21 votes Error rate Expert Labels Cost

Cost vs. Quality MTurk MTurk MTurk 7 votes 11 votes 21 votes Error rate Consensus Agreement Games Expert Labels Cost

Outline The problem Related Work Our method Experimental results

Related Work Image labeling games. The ESP Game. Luis von Ahn and Laura Dabbish. (2004)

Related Work Image labeling games. The ESP Game. Luis von Ahn and Laura Dabbish. (2004) Consensus on graphs. Network graph coloring. Stephen Judd, Michael Kearns and Yevgeniy Vorobeychik. (2010)

Related Work Image labeling games. The ESP Game. Luis von Ahn and Laura Dabbish. (2004) Consensus on graphs. Network graph coloring. Stephen Judd, Michael Kearns and Yevgeniy Vorobeychik. (2010) Peer agreement methods. Peer Truth Serum. Boi Faltings, Pearl Pu, Bao Duy Tran and Radu Jurca. (2014)

Outline The problem Related work Our method Experimental results

Idea

Idea paper glass ?

Idea paper paper glass glass ? ?

Consensus Agreement Games Clique 1 Clique 2 Workers see all clique 1 votes Workers see all clique 2 votes

Consensus Agreement Games Clique 1 Clique 2 Workers see all clique 1 votes Workers see all clique 2 votes Shared image No communication between cliques

Worker Experience Time remaining: 120 seconds Payments: 1/8 cent + ½ cent consensus + 1 ½ cents agreement

Consensus Agreement Games Clique 1 Clique 2 Workers see all clique 1 votes Workers see all clique 2 votes Shared image glass glass glass glass glass glass No communication between cliques Case 1: consensus agreement

Consensus Agreement Games Clique 1 Clique 2 Workers see all clique 1 votes Workers see all clique 2 votes Shared image paper glass glass paper paper glass No communication between cliques Case 2: no consensus

Consensus Agreement Games Clique 1 Clique 2 Workers see all clique 1 votes Workers see all clique 2 votes Shared image paper glass paper paper paper glass No communication between cliques Case 3: single consensus

Consensus Agreement Games Clique 1 Clique 2 Workers see all clique 1 votes Workers see all clique 2 votes Shared image paper glass paper paper glass glass No communication between cliques Case 4: consensus disagreement

Outline The problem Related work Our method Experimental results

OpenSurfaces, material recognition Ceramic or Painted or Ceramic, plastic Plastic or glass? wood? or foliage? glass? http://opensurfaces.cs.cornell.edu/publications/opensurfaces/

Places, scene recognition Bedroom, hotel River, ocean room or nursery? or coast? http://places.csail.mit.edu/

Clique size? versus

OpenSurfaces, cliques of 2 MTurk 7 votes 3 cents, 19% 37% drop Error rate Consensus Agreement Games 12 cents, 12% Cost

Places, cliques of 3 MTurk 7 votes 3 cents, 24% 38% drop Error rate Consensus Agreement Games 15 cents, 15% Cost

Consensus Agreement Games Consensus Agreement Interactive Incentives Payments: Game Peer 1/8 cent + ½ cent consensus + 1 ½ cents agreement

Questions? Thanks for listening!