Ef Effects ects of of I Ince ncentiv ntive e Amou Amount nt and and Typ ype e on on Web b Sur Survey ey Res espon ponse se Ra Rates tes th An Pr Presentation ion at the 69 69 th Annual l AA AAPO POR Co R Conference An Anaheim im, , CA CA May 2014 Jared Coopersmith • Lisa Klein Vogel • Tim Bruursema Kathleen Feeney
Presentation Overview • Introduction – Challenge – Using incentives to increase response rates – Hypotheses – Related work • Methods – Survey and sample description – Experimental conditions • Results • Summary • Discussion 2
Introduction 3
Challenge • The National College Ready Survey (NCRS) is a web-administered survey of school principals – Sponsored by the Bill & Melinda Gates Foundation – Four waves of data collection • Principals face many competing demands on – Personal time and availability – School participation in research • These factors can affect principals’ willingness to respond to survey requests 4
Using Incentives to Increase Response Rates • The NCRS offers significant ($50) post-paid incentives for completion – Increase response – Minimize costs • Wave 2 included an incentive experiment to inform future data collection • Incentives can help increase response rates and sample representativeness, but depends on a number of factors • Web administration poses unique administrative challenges – Cash is more effective than gifts – Pre-paid incentives are most effective, but difficult to administer via web 5
Hypotheses • H1: Offering a differential incentive for completion within the first three weeks of the field period will yield higher response rates both early on and overall • H2: Providing a pre-paid incentive before data collection yield higher response rates due to the “norm of reciprocity” • H3: Providing a pre-paid incentive to nonresponders midway through the field period will yield higher response rates 6
Related Work • Pre-paid incentives – Often more effective than post- paid or “promised” incentives (Singer and Ye 2013; Göritz 2010 ) • Early response or “early bird” incentives – Can be more effective than even pre-paid incentives (LeClere et al. 2012) • Refusal conversion incentives – Some studies point to only using refusal conversion payments (Singer and Ye 2013) – But is this ethically problematic or unfair? (Presser 2008) 7
Incentive Experiment • Based on these hypotheses, we implemented an experiment to compare the relative effectiveness of – $50 post-completion incentive + additional $50 for completion in first three weeks – $50 post-completion incentive + $25 pre-paid incentive • Control condition – $50 post-completion incentive only • Subgroup eligible for $25 refusal conversion incentive Group 1 Group 2 Group 3 Control Standard $50 $50 $50 $50 incentive Additional $50 (early $25 (pre-pay) $25 (refusal None incentive response) conversion) Total possible $100 $75 $75 $50 incentive 8
Methods 9
Data Collection Activities • Mode: Web-only – 39 questions, 15 minutes • Data collection consisted of – Invitation email and postal mailing – Weekly email reminders – Monthly reminder calls to non-completers • Incentive administration: Amazon.com gift card code – Pre-payment sent via email and postal letter • Included Amazon.com $25 gift card code and web address for Amazon.com – Post-payment sent via email • Included gift card code, payment amount, link to Amazon.com, and redemption instructions – Post-payments sent approximately every 8 – 10 days after survey completion 10
Sample Description (1) • Size: n = 2,034 • Composition: Elementary, middle, and high school principals • Selection method: Stratified probability proportional to size (PPS) • Field period: 10/30/13 – 3/31/14 (21 weeks) • Sample was fielded in four different groups during the data collection period – Group 1: n = 1,062; in the field for 21 weeks – Group 2: n = 259; in the field for 17 weeks – Group 3: n = 402 ; in the field for 12 weeks – Group 4: n = 311; in the field for 5.5 weeks 11
Sample Description (2) • $50 post-pay incentive, plus $50 (early $25 $25 (refusal None response) (pre-pay) conversion) (control) Total 560 560 280 280 Sample 523 524 262 258 released 12
Results 13
Results: $100 Early Response Incentive 100 90 80 Early 70 Response Incentive 56.6 60 55.4 50 Post-Pay 40 Only 29.7** 30 20.1 20 10 0 Early Response Cutoff Final Response ** p < 0.05 14
Results: Early Response (cont.) 100 90 80 70 60 Early 50 Response Incentive 40 Post-Pay Only 30 20 10 0 15
Results: $25 Pre-Paid Incentive 100 90 80 70 56.6 60 54.6 Pre-Pay Incentive 50 Post-Pay Only 40 30 20 10 0 Final Response ** p < 0.05 16
Results: $25 Refusal Conversion (Overall) 100 90 80 70 Refusal 56.5 56.7 60 Conversion 50 Incentive Post-Pay Only 40 30 20 10 0 Final Response ** p < 0.05 17
Results: $25 Refusal Conversion (Targeted) 100 90 80 70 60 Refusal Conversion Incentive 50 Post-Pay Only 40 30 24.1* 20 12.9 10 0 Final Response * p < 0.10 18
Results: Time in Field Aver erage ge days days-to to-complete complete Treatment Control Incentive group group Early response 41.8** 48.8 incentive Pre-paid incentive 46.1 48.8 Nonresponse 55.3*** 42.4 conversion incentive *** p < 0.01, ** p < 0.05 19
Summary 20
Overall Impact of Incentives • Early response incentive – Higher response rate within the incentive period • Significant effects overall and among most subgroups – No effect on final response rates • Effect diminished immediately following the incentive period • No difference in response rates by the second half of data collection – Reduced average time in field by 1 week • Pre-paid incentive – No significant effect on response rates or length of time to complete • Post-paid incentive significantly more effective in two subgroups • Refusal conversion incentive – No significant effect on response rates overall – Among those eligible for incentive, did show some evidence of effect – Increased average time to complete 21
Discussion 22
Areas for Further Investigation • Cash vs. electronic incentive administration: Effects of incentive delivery • Incentive amounts: Is bigger always better? • Timing of early response period • Effects on data quality 23
Citations • Göritz, Anja S. “Using Lotteries, Loyalty Points, and Other Incentives to Increase Participant Response and Completion.” In Advanced Methods for Conducting Online Behavioral Research , edited by Samuel D. Gosling and John A. Johnson. Washington, DC: American Psychological Association, 2010. • Gouldner, Alvin W. “The Norm of Reciprocity: A Preliminary Statement.” American Sociological Review, vol. 25, no. 2, 1960, pp. 161 – 78. • LeClere, Felicia, Sheldonn Plummer, Jennifer Vanicek, Ashley Amaya and Kari Carris. “Household Early Bird Incentives: Leveraging Family Influence to Improve Household Response Rates.” Presented at the Joint Statistical Meetings, San Diego, CA, 2012. • Singer, Eleanor, and Cong Ye. “The Use and Effects of Incentives in Surveys.” The Annals of the American Academy of Political and Social Science, vol. 645, 2013, p. 112 – 141. 24
For More Information • Jared Coopersmith JCoopersmith@mathematica-mpr.com • Lisa Klein Vogel LKlein@mathematica-mpr.com • Tim Bruursema TBruursema@mathematica-mpr.com • Kathleen Feeney KFeeney@mathematica-mpr.com 25
Recommend
More recommend
