Everyone On Mechanical Turk is Above a Threshold of Digital Literacy: Using Facebook Ads to Measure Online Media Effects Kevin Munger, Mario Luca, Jonathan Nagler, Joshua Tucker Penn State University and Princeton University September 7, 2018
Overview Online media effects are rapidly changing—how do we keep up?
Overview Online media effects are rapidly changing—how do we keep up? Online media effects are uniquely heterogeneous—how do we generalize, how do we study the correct populations?
Overview Online media effects are rapidly changing—how do we keep up? Online media effects are uniquely heterogeneous—how do we generalize, how do we study the correct populations? Our intuitions about social media can be actively misleading—how do we adjust?
Overview Online media effects are rapidly changing—how do we keep up? Online media effects are uniquely heterogeneous—how do we generalize, how do we study the correct populations? Our intuitions about social media can be actively misleading—how do we adjust? Takeaway from this paper: traditional sampling and traditional survey experiments fail to allow us to study low digital literacy populations
Overview Online media effects are rapidly changing—how do we keep up? Online media effects are uniquely heterogeneous—how do we generalize, how do we study the correct populations? Our intuitions about social media can be actively misleading—how do we adjust? Takeaway from this paper: traditional sampling and traditional survey experiments fail to allow us to study low digital literacy populations Case study: clickbait!
What is clickbait? “Clickbait” is a new term for an old phenomenon.
What is clickbait? “Clickbait” is a new term for an old phenomenon. Media companies’ strategy always determined by technological, political, regulatory contexts
What is clickbait? “Clickbait” is a new term for an old phenomenon. Media companies’ strategy always determined by technological, political, regulatory contexts New technology lowers cost of news production/distribution � new entrants competing for attention
Classic Clickbait
Modern clickbait Various formulations have come and gone; ongoing battle with Facebook
Modern clickbait Various formulations have come and gone; ongoing battle with Facebook Political clickbait is necessarily partisan
Modern clickbait Various formulations have come and gone; ongoing battle with Facebook Political clickbait is necessarily partisan ◮ High levels of affect polarization
Modern clickbait Various formulations have come and gone; ongoing battle with Facebook Political clickbait is necessarily partisan ◮ High levels of affect polarization ◮ Partisans are the biggest consumers of political news
Modern clickbait Various formulations have come and gone; ongoing battle with Facebook Political clickbait is necessarily partisan ◮ High levels of affect polarization ◮ Partisans are the biggest consumers of political news ◮ Signalling partisanship can supplant source cues in establishing credibility
Modern clickbait Various formulations have come and gone; ongoing battle with Facebook Political clickbait is necessarily partisan ◮ High levels of affect polarization ◮ Partisans are the biggest consumers of political news ◮ Signalling partisanship can supplant source cues in establishing credibility One common form of partisan clickbait with potentially damaging consequences: emotional clickbait
Emotional clickbait Turn a partisan headline into emotional clickbait by adding one of these phrases
Emotional clickbait Turn a partisan headline into emotional clickbait by adding one of these phrases ◮ People are loving this:
Emotional clickbait Turn a partisan headline into emotional clickbait by adding one of these phrases ◮ People are loving this: ◮ Democrats are freaking out:
Emotional clickbait Turn a partisan headline into emotional clickbait by adding one of these phrases ◮ People are loving this: ◮ Democrats are freaking out: ◮ This will make you furious:
Emotional clickbait Turn a partisan headline into emotional clickbait by adding one of these phrases ◮ People are loving this: ◮ Democrats are freaking out: ◮ This will make you furious: ◮ Republicans are shocked...
What we tried Online survey experiment
What we tried Online survey experiment Randomly assign respondents to one of four different headlines, keeping the story constant
What we tried Online survey experiment Randomly assign respondents to one of four different headlines, keeping the story constant Look for effets on affective polarization, trust in media and information retention questions
Null Results from First MTurk Study Tried again: made the subject matter more topical, added a placebo condition
Null Results from First MTurk Study Tried again: made the subject matter more topical, added a placebo condition (After the pilot, we pre-registered the R code we used to analyze all results)
Null Results from Second MTurk Study Tried again: shortened the survey, removed “preference for clickbait” questionnaire which could dampen treatment effects
Null Results from Third MTurk Study
Null Results from Third MTurk Study Is Mturk the problem?
Null Results from Third MTurk Study Is Mturk the problem? There were pretty big differences between MTurk and CCES in 2012 (Huff and Tingley, 2015)
Null Results from Third MTurk Study Is Mturk the problem? There were pretty big differences between MTurk and CCES in 2012 (Huff and Tingley, 2015) Many classic (non-digital) experiments replicate on MTurk (Coppock, 2018)
Null Results from Third MTurk Study Is Mturk the problem? There were pretty big differences between MTurk and CCES in 2012 (Huff and Tingley, 2015) Many classic (non-digital) experiments replicate on MTurk (Coppock, 2018) Econ-style experiments also largely replicate on MTurk compared to students or a nationally representative sample (Snowberg and Yariv, 2018)
Null Results from Third MTurk Study Is Mturk the problem? There were pretty big differences between MTurk and CCES in 2012 (Huff and Tingley, 2015) Many classic (non-digital) experiments replicate on MTurk (Coppock, 2018) Econ-style experiments also largely replicate on MTurk compared to students or a nationally representative sample (Snowberg and Yariv, 2018) But: MTurk users are all above a certain threshold of digital literacy
Null Results from Third MTurk Study Is Mturk the problem? There were pretty big differences between MTurk and CCES in 2012 (Huff and Tingley, 2015) Many classic (non-digital) experiments replicate on MTurk (Coppock, 2018) Econ-style experiments also largely replicate on MTurk compared to students or a nationally representative sample (Snowberg and Yariv, 2018) But: MTurk users are all above a certain threshold of digital literacy Actually interested in the effect of clickbait on the clickers (Leeper, 2016; Knox et al., 2014)
The Clickers
Ages of Online Samples Density of Respondents group Facebook MTurk USA 25 50 75 100 Age
Null Results from the FB Study We got the right sample and didn’t find results
Attrition from Online Samples Percentage of Respondents False start Roll Off group Facebook MTurk New Tab DVs
Null Results from the FB Study We got the right sample and didn’t find results Attrition was non-random and covaried with demographics of interest
Ages of FB Sample at Attrition Points Density of Respondents group Finished New Tab Other 25 50 75 Age
Ages of MTurk Sample at Attrition Points Density of Respondents group Finished New Tab Other 25 50 75 100 Age
Examine Predictors of Stopping at New Tab Combine the data, run a fully interacted model to look at differential effects in the two samples
Effect of Age on Stopping at New Tab: MTurk v Facebook ● ●
“Attention Checks” With Digitally Naive Populations Passed attention check MTurk: 82% Passed attention check FB: 57%
Time Spent on Headline Choice Sets Attention Check Time on Stopping at New Tab: MTurk v Facebook ● ●
Time Spent on Headline Choice Sets Attention Check Time on Stopping at New Tab: MTurk v Facebook ● ● Placebo Choice Time on Stopping at New Tab: MTurk v Facebook ● ●
Attention Checks Combine the data, run a fully interacted model to predict missing the attention check
Attention Checks Combine the data, run a fully interacted model to predict missing the attention check Differential effects (similar to above)
Attention Checks Combine the data, run a fully interacted model to predict missing the attention check Differential effects (similar to above) Effect of Age on Missing Attention Check: MTurk v Facebook ● ●
Attention Checks Combine the data, run a fully interacted model to predict missing the attention check Differential effects (similar to above) Effect of Age on Missing Attention Check: MTurk v Facebook ● ● Conditional on all covariates, “effect” of being in the Facebook sample is negative and significant (p < . 05)
Recommend
More recommend
