Mobile Web Survey Design: Scrolling versus Paging, SMS versus - PowerPoint PPT Presentation

Mobile Web Survey Design: Scrolling versus Paging, SMS versus E-mail Invitations DC-AAPOR & WSS Summer Conference Preview/Review 2014 Aigul Mavletova, NRU Higher School of Economics, Russia Mick P. Couper, Survey Research Center, University of Michigan July 22-23, 2014 1 Washington DC

Paging vs. Scrolling Design AAPOR suggestion: Optimizing an online survey for completion in a mobile browser is something of an art, but there are several notable conventions that can make it easier for respondents to complete. The overall layout should minimize the need for complete. The overall layout should minimize the need for scrolling (either horizontally or vertically) to the extent possible. The number of questions per screen should generally be two or less to minimize scrolling. Mobile Technologies for Conducting, Augmenting and Potentially Replacing Surveys: Report of the AAPOR Task Force on Emerging Technologies in Public Opinion Research (April, 2014) 2

Mobile web vs. PC web surveys � Higher breakoff rates in mobile web surveys: 2-12 times higher. � Longer Longer completion completion times: times: 1.5-3 1.5-3 times times longer in mobile web surveys. 3

Paging vs. Scrolling Design PC web surveys: • No differences in response rates and breakoff rates. • Simple skip logic: a scrolling version takes less time. More complex skip logic: a scrolling version takes more time. • A scrolling design increases item nonresponse rates. • A scrolling design increases item nonresponse rates. (Peytchev et al. 2006; Tourangeau et al. 2004; Vehovar et al., 2000) Mobile web surveys: • McGeeney & Marlar (2013): higher breakoff rates in the paging version (13% of mobile respondents; a 13-item survey). 4

SMS vs. E-mail • E-mail produces higher response rates than an SMS invitation in PC web surveys (Bosnjak et al. 2008). • SMS significantly increases the percentage of mobile web respondents (Crawford et al. 2013). 5

Hypotheses and Experimental Design 6

Hypotheses Hypotheses Lower breakoff rate, longer completion times, and Scrolling vs. lower rate of item nonresponse are expected in the Paging paging design. 7

Hypotheses Hypotheses Lower breakoff rate, longer completion times, and Scrolling vs. lower rate of item nonresponse are expected in the Paging paging design. SM S will result in lower participation rates. SM S will result in lower participation rates. SM S vs. SM S vs. E-mail invitations are expected to increase the E-mail percentage of PC web respondents. Changing the mode of the reminder will improve SM S vs. participation rates compared to the conditions which E-mail have the invitation and reminder in the same mode. 8

Experimental Design Survey Design Paging Scrolling 17 screens 2 screens 9

Experimental Design Survey Invitation- Design Reminder Paging Scrolling Invitation Reminder E-mail E-mail 17 screens 2 screens SM S E-mail without URL SM S + + + E-mail + + - 10

Data Collection � Volunteer online access panel (Online Market Intelligence). � 7 minutes survey. � Respondents were encouraged to complete the survey via mobile devices. via mobile devices. � Fieldwork: 9-29 th April, 2013, Russia. � Software: Unipark. � 2,110 respondents: 4, 000 invitations, participation rate=52.8% (1 invitation, 1 reminder) 11

Questionnaire � 17 questions: volunteer activities, practices of helping other people, mobile web usage patterns, evaluation of the survey. � No screening questions, quotas, and skipping logic. � All questions were not obligatory. 12

Results: Scrolling vs. Paging 13

Participation Rates 55,0% 53,8% 54,0% 53,0% 51,8% 52,0% 51,0% 49,9% 48,6% 50,0% Scrolling 49,0% Paging Paging 48,0% 47,0% 46,0% Participation rate Participation rate (mobile only) Number of completed interviews: 1,075 in the scrolling and 1,035 in the paging version 14

Sample Composition by Devices Device % Smartphones 73.7% Feature phones 9.1% Tablets Tablets 3.3% 3.3% PC 13.8% Total 2,110 No difference between the scrolling and paging design 15

Breakoff Rates 12,0% 10,2% 10,0% 8,0% 6,9%* * * 8,0% 6,0%* * 6,0% 3,3%* * Scrolling 4,0% 2,1%* * * 2,1%* * * 2,0% Paging Paging 0,0% Breakoff rate Breakoff rate on Breakoff rate on the first page the subsequent p =0.067 pages 16 Chi-square ( df =1), ** p <0.01, *** p <0.001

Breakoff Rate by Devices and Operating Systems Device* * * Device* * * Feature phones 30,3% Chi-square= Smartphones 6,5% 171.69*** (df =3) PC 5,5% Tablets 2,8% 0,0% 10,0% 20,0% 30,0% 40,0% Operating systems* * * Symbian 12,8% Android 5,9% Chi-square = 21.35*** Windows Phone 5,4% (df=4) Bada 4,3% iOS 2,2% 17 *** p <0.001 0,0% 2,0% 4,0% 6,0% 8,0% 10,0% 12,0% 14,0% 16,0%

Completion Times (minutes) 10 9,1* * * 9 8 6,6* * * 7 6 4,9* * * 4,5* * * 5 Scrolling 4 3 Paging Paging 2 1 0 Objective mean Subjective evaluation of completion time completion time 18 t-test, *** p <0.001

Technical Difficulties Chi-square Scrolling Paging Device ( df =1) Any technical problems 18.9% 25.8% while completing the while completing the 14.56* * * 14.56* * * (203) (203) (267) (267) survey 19 *** p <0.001

I tem Nonresponse Device Scrolling Paging Statistics At least one item Chi-square nonresponse 8.4% 9.9% =1.40 (nonrespondents) (df=1) 1.84% 1.44% t=1.12 Overall item nonresponse (SD=8.6%) (SD=7.8%) (df=2,108) 20

SM S vs. E-mail 21

Participation Rates 60,0% 52,8% 52,7% 51,3%* * * 50,0% 43,7%* * * 40,0% 30,0% SM S SM S 20,0% E-mail 10,0% 0,0% Participation Rate Participation Rate (mobile devices only) 22 Chi-square ( df =1), *** p <0.001

Participation Rates 70,0% 60,9% 60,0% 54,6% 53,2% 48,7% 50,0% 44,1% 40,0% 30,0% 20,0% 10,0% 0,0% E-mail-SM S SM S-E-mail SM S-SM S SM S-E-mail E-mail-E-mail without URL 23 Chi-square = 39.316 (df=4), p <0.001

Participation Rate (Mobile Only) 60,0% 53,5% 52,4% 52,1% 47,7% 50,0% 40,0% 32,8% 30,0% 20,0% 10,0% 0,0% E-mail-SM S SM S-SM S SM S-E-mail SM S-E-mail E-mail-E-mail without URL 24 Chi-square = 62.039 (df=4), p <0.001

PC Respondents 45,0% 40,3% 40,0% 35,0% 27,0% 30,0% 25,0% 20,0% 20,0% 15,0% 10,2% 10,0% 3,8% 3,1% 5,0% 0,0% E-mail-E-mail E-mail-SM S SM S-E-mail SM S-E-mail SM S-SM S without URL 25

Progress of Participation Rates Participation Rate 60% 53% 50% 37% 36% 40% 27% 31% 33% 35% 27% 31% SM S SM S 26% 26% 27% 27% 30% 19%21% 24% E-mail 21% 15% 20% 11% 10% Hours 0% Total 0 1 2 3 4 5 6 7 8 9 10 26

Main Findings (1) Scrolling design makes the process of completing the survey easier. (2) SMS invitation is more efficient compared to e- mail in encouraging respondents to complete the mail in encouraging respondents to complete the survey via a mobile device. 27