Planning to Control Crowd-Sourced Workflows Daniel S. Weld - PowerPoint PPT Presentation

Planning to Control Crowd-Sourced Workflows Daniel S. Weld University of Washington (Joint Work with Jonathan Bragg, Lydia Chilton, Peng Dai, Shih-Wen Huang, James Landay, Chris Lin, Angli Liu, Andrey Kolobov, Mausam & Stephen Soderland)

James Shih-Wen Jonathan Lydia Peng Chris Huang Landay Bragg Chilton Dai Lin Thanks Stephen Angli Andrey Mausam Liu Kolobov Soderland 3

Crowdsourcing • Performing work by soliciting effort from many people • Combining the efforts of volunteers/part-time workers (each contributing a small portion) to produce a large or significant result 4

Crowdsourcing Successes 190 M reviews of 4.4 M businesses Answers to 7.1 M prog. questions Universal reference for anything 5

Citizen Science 800,000 volunteers – Hubble images Discovered “Hanny’s Voorwerp” black-hole “Pea galaxies” Crowdsourced bird count & identification Migration shift -> effect of climate change Game to find 3D structure of proteins. Solved 15 year outstanding AIDS puzzle 6

Labor Marketplaces Will Grow to $5B by 2018 [Staffing Industry Analysts] • 2.7 million workers • 540,000 requestors • 35M hours worked in 2012 60% Growth Hours / week Nationality Charts from Panos Ipierotis’ blog; phone from pixabay . Nationality and gender of 7

AI in Crowdsourcing Collective assessment mean field approximation belief propagation variational inference independent Bayes classifier hierarchical clustering Bayesian bias mitigation Mallows model two coin model Bayes net Bayesian aggregation majority vote Chinese restaurant process HybridConfusion EM gold questions ordinal-discrete mixture model multidimensional wisdom of crowds temporal likelihood mutual information minimax conditional entropy convex objective function open source datasets 8

AI in Crowdsourcing • Collective assessment – State estimation/tracking – passive 9

The Rest of Crowdsourcing • control of simple tasks – optimize redundancy for best quality-cost tradeoff • complex tasks – optimize workflows; pick the BEST one? A-B Testing • task routing – finding the right workers for the right job • make workers skilled – training; when? how much? 10

Iterative Improvement Workflow initial artifact [Little et al, 2010]

Iterative Improvement Workflow initial artifact [Little et al, 2010]

Iterative Improvement Workflow initial artifact [Little et al, 2010]

Iterative Improvement Workflow [Little et al, 2010] First version Version after 8 iterations A parial view of a pocket A CASIO multi-function, solar powered calculator together with some scientific calculator. coins and a pen. A blue ball point pen with a blue rubber grip and the tip extended. Six British coins; two of £1 value, three of 20p value and one of 1p value. Seems to be a theme illustration for a brochure or document cover treating finance - probably personal finance. 23

[Little et al, 2010] 24

25 [Little et al, 2010]

Controller for a Task Generate a Task Action job b submit output Controller 26

Artificial Intelligence 101 Agent Sensors Percepts Environment POMDP ? Actions Actuators 27

Partially-Observable Markov Decision Process Input: World State Actions Observe: Next State s’ = <x’, y’> s = <x, y> P(s’ | s, a) Reward = f(s, a, s’) Cost c Output: Construct policy , π : S à A, that chooses best action for each state I.e., actions that maximize expected reward – costs over time While learning action & reward probabilities (Reinforcement learning) Figure from Dan Klein & Pieter Abbeel - UC Berkeley CS188: http://ai.berkeley.edu.] 28 And Sarah Reeves (http://dear-theophilus.deviantart.com/)

Partially-Observable Markov Decision Process Input: Belief State Actions Observe: Noisy Sensor = f(s’) P(s) P(s’ | s, a) Reward Cost c Output: Construct policy , π : S à A, that chooses best action for each state I.e., actions that maximize expected reward – costs over time While learning action & reward probabilities (Reinforcement learning) Figure from Dan Klein & Pieter Abbeel - UC Berkeley CS188: http://ai.berkeley.edu.] 29

Solving the POMDP Constructing the policy , π, to choose the best action • Many algorithms – Point-based methods – UCT on discretized space – Lookahead search with beta distribution belief states Q * (s, a) = Σ s’ P(s’ | s, a) [ R(s, a, s’) + γ Max a Q * (s, a) ] • Exploration / exploitation problem – ε -greedy – UCB / Multi-armed bandit 30

From To (Hidden) <x,y> coords World State Actions ? Move Grasp Costs Power used Reward Robot figure from Dan Klein & Pieter Abbeel - UC Berkeley CS188: http://ai.berkeley.edu.]

From To (Hidden) <x,y> coords Quality Q 1 , Q 2 ∈ (0,1) World State Actions Move Improve caption task Grasp Vote best caption Costs Power used $$ paid to workers F(quality returned) Reward Robot figure from Dan Klein & Pieter Abbeel - UC Berkeley CS188: http://ai.berkeley.edu.] 32

Comparison [Dai, Lin, Mausam, Weld AIJ’13] 0.8 0.75 POMDP Hand Coded 0.7 Quality 0.65 0.6 0.55 0.5 40 images, same average cost Controlling quality: POMDP 30% cheaper 33

Allocation of Human Labor POMDP Hand Coded

Human Labor Redirected POMDP Hand Coded