An Analytic Framework for Human Computation Crowdsourcing and Human Computation Instructor: Chris Callison-Burch Website: crowdsourcing-class.org
Today’s topic
Classification System for Human Computation • Motivation • Quality Control • Aggregation • Human Skill • Process Order • Task-request Cardinality
Motivation How can we motivate people to participate? Even with a low barrier to entry (anyone with an computer can contribute) we still need to make a case why they should contribute.
Motivation: Pay • Easiest way to recruit workers. • Downside: provides incentive to cheat • Problem might be exacerbated when the crowd workers are anonymous • MTurk uses micropayments • Online temping services provide higher wages: LiveOps, ODesk, etc • CrowdFlower tried non-monetary payments (virtual goods and currencies, SwagBucks)
Motivation: Altruism • People want to do good • When Jim Gray went missing, volunteers searched 500k satellite images • After the Haitian earthquake, diaspora translated 1000 messages per day
Eske lekol kolej marie anne Was the College Marie Anne kraze?mesi school destroyed? Thank you. We can’t prevent the Nou pa ka anpeche moustik yo mosquitoes from biting because mo`de nou paske yo anpil. there are so many. Please heart is breaking tanpri ke`m ap kase mwen pa because I have no news of my ka pran nouvel manmanm. mother. 4636: The Doctors without 4636:OpitalMedesensanFwontie Borders Hospital in Delmas 19 is `delmas19lafe`men. Opital sen closed. The Saint Louis lwi gonzag nan delma 33 pran Gonzaga hospital in Delmas 33 an chaj gratwit- man tout moun is taking in sick and wounded ki malad ou blese people for free
Motivation: Reputation • Sometimes people will contribute in order to build their profile within a community • Example: stackoverflow
Motivation: Enjoyment • Games with a purpose is a strategy to try to make a task fun • In the ESP game two players look at an image and try to guess what words the other is thinking • In doing so they label images on the web
Luis Von Ahn == Tom Sawyer
Motivation: Implicit Work • It is sometimes possible to make people do work alongside some other task
Motivation • Pay • Altruism • Reputation • Enjoyment • Implicit Work • Can you think of others?
Quality Control Even if people are motivated to participate, how do we know that they are doing work conscientiously? Can we trust them not to cheat or sabotage the system? Even if they are acting in good faith, how do we know that they’re doing things right?
Quality Control: Reputation Check • Mechanical Turk uses a reputation system • When a Turker submits poor work, Requesters reject it • The Turker’s approval rate is displayed to all other Requesters
QC: Agreement and Redundancy • The ESP game uses the labels that two players independently agree on • Similar technique is often used in MTurk, when each item is done independently • Redundancy allows a voting on ambiuous answers / opinions • It is also helpful for identifying workers who are consistently divergent
QC: Gold Standard • In MTurk we commonly mix in questions with a known answer alongside new questions • This is similar to agreement, but now we check agreement against experts or trusted workers • For multiple choice questions, gold standard allows for automatic grading
QC: Second-pass review • Do second-pass grading when gold standard don’t allow automatic grading • Often times the second-pass HIT can be automatically gradable • This makes the whole pipeline fully automated and ensures high quailty
• Why was Heather Was arrested actress Heather . Medikamentes unknown have the Actress Heather Locklear was due to Heather Locklear Arrested for Locklear arrested? Heather Locklear Arrested for driving under the influence of an Locklear because of the driving under effect of a fahrens under actress driving under the influence of drugs ‣ She was arrested on driving under the influence of drugs heather locklear arrested the effect of an unknown medicine unknown drug arrested suspicion of driving under The actress Heather Locklear that In Santa. One is, melrose place Actress Heather Locklear, by the The actress Heather Locklear, The actress Heather Locklear, the influence of drugs. is known to the Amanda through role of Amanda from the series the series of the role of the Amanda of the popular series Amanda of the popular series 'remember the locklear actress the "Melrose Place" is known, was this the role from the series "Melrose Melrose Place, was arrested this Melrose Place, was arrested this weekend in Santa Barbara heather this weekend, because of Place" was arrested at this weekend in Santa Barbara weekend in Santa Barbara weekend in Santa Barbara the fahrens Barbara (California) in (California) because of driving (California) after driving under the (California) after driving under the Driving under the influence (Californium) because of the under the influence of an unknown effect unknown medikamentes influence of drugs. A witness influence of drugs. A witness driving under the effect of an drug arrested. A witness had arrested People 'magazine. The viewed her performing viewed her performing traffic police California, observed how it quite strange way unknown medicine. A female inappropriate maneuvers while inappropriate maneuvers while tried to park their extended gap in spokesman for the auszufahren witness observed she attempted trying to take her car out from a trying to take her car out from a Driving while medicated montecito reported in its way from Montecito, reported spokesman in quite strange way how to go parking in Montecito, as revealed parking in Montecito, as revealed from their parking space in for the traffic police from California tried parklücke type strange right, to People magazine by a to People magazine by a Montecito, speaker of the traffic to the magazine `People '. The you have seen as a witness. . In spokesman for the Californian spokesman for the Californian witness told in detail that Locklear some Zeitung, as and when they police of californium told the Highway Police. The witness Highway Police. The witness DUI `after 16:30 clock durchdrückte warehouse `People'. The female tried to a great deal of 30 p.m., stated that around 4.30pm Ms. stated that around 4.30pm Ms. pedal and a lot of noise made by witness told in detail, that Locklear witness the detail of history Locklear "hit the accelerator very Locklear "hit the accelerator very locklear after 16: that trying to her car to the rear or front 'pressed `after 16:30 clock violently, making excessive noise violently, making excessive noise of the park gap to move, and accelerator and a lot of noise did durchdrückte peddle noise and its while trying to take her car out while trying to take her car out Driving while using drugs when she attempted to move their when she went backwards, took it progress was made parklücke for from the parking with abrupt back from the parking with abrupt back car towards behind or forward a few times in their Sunglass'. the car or moving backwards, they and forth maneuvers. While and forth maneuvers. While had they times of their from the parking space, and when Somewhat later the witness saw reversing, she passed several reversing, she passed several it went backwards, she pulled sonnenbrille ' . The first was the beginning of the actress times in front of his sunglasses." times in front of his sunglasses." Medikamentes itself together unites Male at their probably had not recognized that probably recognised that locklear Shortly after, the witness, who, in Shortly after, the witness, who, in sunglasses'. A little later the Locklear on a nearby road and a nearby road and anhielt, had a first time, apparently had not a first time, apparently had not female witness that did probably not, with the witness to the car off stopped the car exit. recognized the actress, saw Ms. recognized the actress, saw Ms.
QC: Defensive task design • Try to design tasks so that they are nearly as hard to cheat as they are to complete • For my translation HITs, people frequently would paste text into Google Translate • I converted the text into images, then people had to transcribe it.
QC: Statistical models • Sometimes it is possible to have prior knowledge about what the range of expected answers should be • Use your statistical knowledge to throw out outliers
QC: Economic incentives • When money is the motivator, it may be possible to use different incentive structures to illicit better results • Pay people more when they reach a certain level of mastery, or when their output passes second pass reviews • CastingWords uses bonuses to do this
Quality Control • Reputation check • Agreement and redundancy • Gold standard + automatic reviewing • Multi-level review • Defensive task design • Statistical filtering • Economic incentives • Others?
Aggregation Part of the process of human computation is to combine all contributions to solve a global problem. The class of problem may determine what strategy is best.
Recommend
More recommend
