Measuring Workers Pre-task Interactions by Jason T. Jacques - PowerPoint PPT Presentation

Measuring Workers’ Pre-task Interactions by Jason T. Jacques supervised by Per Ola Kristensson jtj21@cam.ac.uk people.ds.cam.ac.uk/jtj21

Spotting Advertising Color Outline 100 US-based participants 200 US-based participants 10¢ per study 12¢ per study Q4 2012 Q3 2014

Color Outline Did paying more entice user participation?

Color Participants: 100 ? Time taken: 411 hours Task uptake: every 4h 07m Outline Participants: 200 Time taken: 1113 hours Task uptake: every 5h 34m

06:00 UTC http://aa.usno.navy.mil/data/docs/earthview.php

18:00 UTC http://aa.usno.navy.mil/data/docs/earthview.php

Color ? Workforce: 650,000 † Participants: 100 Task uptake: 0.015% Outline Workforce: 825,000 † Participants: 200 Task uptake: 0.024% † Estimated based on a linear trend of 100,000 crowd workers in March 2007 (http://goo.gl/opQH10); 400,000 in September 2010 (http://goo.gl/G1g9AC); and 500,000 in January 2011 (http://goo.gl/vno0GW).

Color Previews: 388 Participants: 100 Task uptake: 25.8% Outline Previews: 821 Participants: 200 Task uptake: 24.7%

turkmill http://tr.im/turkmill $ turkmill Usage: turkmill [-updategeoip] [-novisitors] [-postback] webpath logfile ... $ turkmill web/path/to/hit /var/log/apache2/access_log /var/log/apache2/access_log2 Worker Preview Complete Conversion Rate 1 09:48 09:48 1÷1 = 1.00 2 10:12 10:13 2÷2 = 1.00 3 10:17 2÷3 = 0.66 4 11:31 11:31 3÷4 = 0.75 5 11:06 11:07 4÷5 = 0.80 6 12:22 12:23 5÷6 = 0.83

Color Outline 0.50 0.45 0.40 0.35 Proportion 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Proportion of Time Elapsed

Conversion Rate Previews Completions 1.00 0.90 0.80 0.70 Proportion 0.60 0.50 0.40 0.30 0.20 0.10 0.00 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Proportion of Time Elapsed Conversion rate graph for Color

Conversion Rate Previews Completions A B C 1.00 0.90 0.80 0.70 Proportion 0.60 0.50 0.40 0.30 0.20 0.10 0.00 0 48 96 144 192 240 288 336 384 432 480 528 576 Hours Passed Demographic study

Color Outline 0.50 0.45 0.40 0.35 Proportion 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Proportion of Time Elapsed

Color Conversion rate (final): 25.8% Nominal conversion rate: 19.5% Completion time: 411 hours Outline Conversion rate (final): 24.4% Nominal conversion rate : 25.8% Completion time: 1113 hours Expected time at 10¢: 1473 hours

Baseline

Without branding

Without value proposition

One page presentation

US-based workers

Non-US workers

Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets Jakob Rogstadius a , Vassilis Kostakos a , Aniket Kittur b , Boris Smus a , Jim Laredo c , Maja Vukovic c a Madeira Interactive Technologies Institute b Carnegie Mellon University c IBM T.J. Watson Research Center University of Madeira 5000 Forbes Avenue Hawthorne NY 10532, USA 9000390 Funchal, Portugal Pittsburgh, PA 15213, USA {laredoj,maja}@us.ibm.com {jakob,vk}@m-iti.org nkittur@cs.cmu.edu tion, audio transcribing and various types of surveys. In Abstract Blood cell return, the people who carry out the work are paid money COUNT these, including partially visible cells Crowdsourced labor markets represent a powerful new pa- for each completed task, often in small amounts: tagging radigm for accomplishing work. Understanding the motivat- Malaria parasite in ring-form with double chromatin dots. an image, for example, may pay a few cents. ing factors that lead to high quality work could have signifi- COUNT these. Crowdsourcing work involves a number of challenges cant benefits. However, researchers have so far found that Malaria parasite in other growth stage. motivating factors such as increased monetary reward gen- different from those faced in traditional work settings. IGNORE these. erally increase workers’ willingness to accept a task or the Crowd workers in general purpose markets like MTurk speed at which a task is completed, but do not improve the may have highly varying expertise, skills, and motivations. Figure 1. Instructions given to participants on how to complete the quality of the work. We hypothesize that factors that in- Employers (“requesters” in MTurk) have very little visibil- experimental task. crease the intrinsic motivation of a task – such as framing a ity into these characteristics, especially compared to a task as helping others – may succeed in improving output traditional organization in which workers are vetted during quality where extrinsic motivators such as increased pay do recruitment, have work histories, have reputations within not. In this paper we present an experiment testing this hy- and outside the organization, and may go through organi- pothesis along with a novel experimental design that enables zational socialization methods such as training to ensure controlled experimentation with intrinsic and extrinsic mo- they can appropriately satisfy their job requirements. Fur- tivators in Amazon’s Mechanical Turk, a popular crow d- sourcing task market. Results suggest that intrinsic motiva- thermore, workers can easily return work for a given job tion can indeed improve the quality of workers’ output, co n- with no repercussions or even create an entirely new pro- firming our hypothesis. Furthermore, we find a synergistic file with a clear reputation. These challenges mean that interaction between intrinsic and extrinsic motivators that employers have more limited means of eliciting high quali- runs contrary to previous lit erature suggesting “crowding ty output than in traditional organizations. out” effects. Our results have significant practical and the o- This study experimentally assesses the interaction of ex- retical implications for crowd work. trinsic and intrinsic motivators in crowdsourcing markets using a novel experimental methodology that controls for Figure 2. A sample image of medium complexity from the experi- mental task.

Global Health Council (high intrinsic motivation)

No sponsor (medium intrinsic motivation)

Rimek International (low intrinsic motivation)

None (Medium) Rimek (Low) GHC (High) 0.50 0.45 0.40 0.35 Conversion Rate 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Proportion of Time Elapsed

http://tr.im/turkmill Measuring Workers’ Pre-task Interactions by Jason T. Jacques supervised by Per Ola Kristensson jtj21@cam.ac.uk people.ds.cam.ac.uk/jtj21