AAPOR 2011 Membership Survey Preliminary Paradata Analysis Heather Hammer, Abt SRBI American Association for Public Opinion Research 67 th Annual Conference Presentation May 18, 2012
Background for the Analysis • Was Senior Study Director at the Temple University Institute for Survey Research (ISR) when the opportunity to conduct the 2011 AAPOR Membership Survey was announced – ISR was a CASES shop – CASES recently added web survey capacity – Wanted to develop and pilot a unified case history coding scheme for ISR’s multi-mode studies – Proposed to add telephone outreach to the 2011 AAPOR Membership Survey e-mail and mail nonresponse effort for those who did not complete a web or mail survey by the third e-mail reminder 2 AAPOR 2011 Membership Survey Paradata Analysis
Background for the Analysis • Relied heavily on David Chearo and Martha Van Haitsma’s Survey Practice paper titled “Standardized Attempt Codes for Unified Multi-Mode Case Histories” (February 2012) • Developed unified incoming vs. outbound categories and codes to record the mode, source, and type of outcome for a maximum of 13 outcomes • Created a data entry screen for interviewers to capture the telephone and other manual contact attempt data 3 AAPOR 2011 Membership Survey Paradata Analysis
Contact Attempt Modes 00 no attempt - action needed 01 in-person/CAPI 02 telephone/CATI/IVR 03 mail 04 web form 05 email 06 fax 07 text message 08 instant message 09 voice message 4 AAPOR 2011 Membership Survey Paradata Analysis
Contact Attempt Sources 1 selected respondent 2 eligible proxy 3 informant 4 interviewer 5 supervisor 6 tracer 7 system generated 5 AAPOR 2011 Membership Survey Paradata Analysis
Selected Outbound Contact with R 056 057 alternate mode requested/needed 058 alternate language requested/needed 059 060 appointment made/request for definite return contact 061 request callback preferred times/days or general 062 request for information or materials 063 suspend contact attempts until future date (e.g., R on vacation) 064 065 promise to return contact/complete survey 066 067 break-off (partial), follow up needed 068 069 6 AAPOR 2011 Membership Survey Paradata Analysis
Outbound Phone Noncontact Examples 050 continuous ring 051 no message left 052 no message possible 053 message left 054 line busy 7 AAPOR 2011 Membership Survey Paradata Analysis
Manual Data Entry Screen 8 AAPOR 2011 Membership Survey Paradata Analysis
Mode and Source Drop Down Lists 9 AAPOR 2011 Membership Survey Paradata Analysis
Data Collection ( June 27 - August 25, 2011) Mean Days Number of Outcome Contact Attempt Completed by Between Individuals in Number Sequence Any Mode t and t-1 Sequence 0 2,844 Total Listed Start 1 2,771 E-mail Invitation 8 2 2,119 E-mail Reminder 1 482 8 3 1,862 E-mail Reminder 2 249 7 4 1,691 Mail Questionnaire 156 2 5 1,640 E-mail Reminder 3 12 5 6 1,348 Outbound Telephone 289 12 7 814 Outbound Telephone 189 8 8 50 Outbound Telephone 88 5 9 7 Outbound Telephone 35 10 0 Outbound Telephone 19 11 1 Outbound Telephone 8 12 0 Outbound Telephone 1 End 13 0 Outbound Telephone 4 10 AAPOR 2011 Membership Survey Paradata Analysis
Completed Surveys by Mode and Type Type as Type as Completion Mode as Percent of All Percent of Percent of Type of Completes All Total Complete Completes Eligible Web Phone Mail Total Total Current 1059 3 145 1207 1986 Members 87.7% 0.2% 12.0% 78.8% 69.8% Former 154 1 31 186 607 Members 82.8% 0.5% 16.7% 12.1% 21.3% Conference 111 1 26 138 251 Attendees 80.4% 0.7% 18.8% 9.0% 8.8% Total 1324 5 202 1531 2844 86.5% 0.3% 13.2% 100.0% 100.0% 11 AAPOR 2011 Membership Survey Paradata Analysis
Cumulative Completes by Mode All Completed Surveys Web Completes Mail Completes Phone Completes N=1,531 N=1,324 N=202 N=5 Completed Typical Survey Sequence N % Cum. % N % Cum. % N % Cum. % N % Cum. % Outcome Outcome Number 1 E-mail invite 0 0.0 0.0 0 0.0 0.0 0 0.0 0.0 0 0.0 0.0 2 E-mail reminder 1 482 31.5 31.5 474 35.8 35.8 8 4.0 4.0 0 0.0 0.0 3 E-mail reminder 2 249 16.3 47.7 245 18.5 54.3 4 2.0 5.9 0 0.0 0.0 4 Mail questionnaire 155 10.1 57.9 152 11.5 65.8 3 1.5 7.4 0 0.0 0.0 5 E-mail reminder 3 12 0.8 58.7 12 0.9 66.7 0 0.0 7.4 0 0.0 0.0 Mostly outbound 6 289 18.9 77.5 241 18.2 84.9 47 23.2 30.7 1 20.0 20.0 phone Mostly outbound 7 189 12.3 89.9 116 8.8 93.7 72 35.6 66.3 1 20.0 40.0 phone Mixed outbound 8 88 5.7 95.6 37 2.8 96.5 48 23.8 90.1 3 60.0 100.0 phone 9 Varies 35 2.3 97.9 23 1.7 98.2 12 5.9 96.0 0 0.0 10 Varies 19 1.2 99.2 15 1.1 99.3 4 2.0 98.0 0 0.0 11 Varies 8 0.5 99.7 7 0.5 99.8 1 0.5 98.5 0 0.0 12 Varies 1 0.1 99.7 0 0.0 99.8 1 0.5 99.0 0 0.0 13 Varies 4 0.3 100.0 2 0.3 100.0 2 1.0 100.0 0 0.0 Total 1,531 100.0 1,324 100.0 202 100.0 5 100.0 12 AAPOR 2011 Membership Survey Paradata Analysis
Cumulative Response Rates by Type 13 AAPOR 2011 Membership Survey Paradata Analysis
Phone Effort by Date for Outcomes 6-9 14 AAPOR 2011 Membership Survey Paradata Analysis
Targeted Nonrespondent Completes • 41.3% of the 1,531 completed surveys, including 33.3% of the 1,324 completed web surveys, 92.6% of the 202 completed mail surveys, and all five of the completed telephone surveys were done between the start of the telephone nonresponse effort at Outcome 6 and the end of data collection at Outcome 13 • Whereas two-thirds (66.7%) of all completed web surveys were done by the time the telephone nonresponse effort was launched at outcome 6, only 7.4% of the completed mail surveys had been received, and no telephone surveys had been completed even though there was telephone contact with some individuals prior to Outcome 6 15 AAPOR 2011 Membership Survey Paradata Analysis
Question for the Paradata Analysis • Did the telephone effort make a meaningful contribution to the response rate? 16 AAPOR 2011 Membership Survey Paradata Analysis
Phone Targeted Nonrespondents Type as Percent Completion Mode as Percent of All of All Targeted Targeted Nonrespondent Type of Nonrespondent Completes Complete Completes Web Phone Mail Total Current 336 3 132 471 Members 71.3% 0.6% 28.0% 74.5% Former 69 1 29 99 Members 69.7% 1.0% 29.3% 15.7% Conference 36 1 25 62 Attendees 58.1% 1.6% 40.3% 9.8% Total 441 5 186 632 69.8% 0.8% 29.4% 100.0% 17 AAPOR 2011 Membership Survey Paradata Analysis
Completed at t+1 after Phone at t Type as Completion Mode as Percent of Percent of Total Type of Total Completes Complete p=0.002 Completes Web Mail Total The 259 web and mail surveys Current 115 81 196 completed at the Members 58.7% 41.3% 75.7% outcome immediately Former 14 18 32 following the Members outbound phone 43.8% 56.3% 12.3% contact or message comprise 41% of all Conference 8 23 31 632 completes Attendees 58.1% 40.3% 12.0% among phone targeted nonrespondents. Total 137 122 259 52.9% 47.1% 100.0% 18 AAPOR 2011 Membership Survey Paradata Analysis
Paradata Analysis Answer to Question • The paradata suggest that the telephone effort boosted the response rate among targeted nonrespondents, however; there was no randomization, no comparison group, and this simple analysis did not model the cumulative effect of mailing the questionnaire, sending a third e-mail reminder, and the telephone effort. – e.g., indirect effects are evident in patterns where telephone contact at time t leads to sending the e-mail link again and a complete at outcome t+2 19 AAPOR 2011 Membership Survey Paradata Analysis
Cumulative Means by Date, Members 20 AAPOR 2011 Membership Survey Paradata Analysis
Cumulative Means, Former Members 21 AAPOR 2011 Membership Survey Paradata Analysis
Thank You • The AAPOR Membership and Chapter Relations Committee • Joe Murphy, RTI International • Liz Hamel, The Henry J. Kaiser Family Foundation • Chase H. Harrison, Harvard University Business School • Dan Merkle, ABC News • Andy Weiss, Abt SRBI • Rene Bautista, NORC 22 AAPOR 2011 Membership Survey Paradata Analysis
For More Information • AAPOR 2011 Membership Survey Final Report • Available at www.aapor.org • Member Connect > Membership Research 23 AAPOR 2011 Membership Survey Paradata Analysis
24 AAPOR 2011 Membership Survey Paradata Analysis
Recommend
More recommend
