Contact Strategies in F2F and Telephone Surveys Steve Schwarzer, - PowerPoint PPT Presentation

Worth the effort? An Examination of Re- Contact Strategies in F2F and Telephone Surveys Steve Schwarzer, Patrick Moynihan, Martha McRoy CSDI Workshop, Limerick, Ireland | March 27, 2018

Research Area • Declining respons onse e rates general ally ly seen as an indicat ation for less qu quali lity, , A lot of effort goes into maintaining response rates • • Non-re respon sponse e is seen as one cause of survey y error and bias • Non-re respon sponse e bias as the difference nce bet etween estimates s for survey y respon pondent ents s and overal all l popu pulat latio ion • Increases of there is a relationship between study subject and likelihood of responding • Inverse of response rate Ho How do survey estima mates es change ge with differen erent t level els s of fieldw dwor ork k effor ort t (re relati tive and absol solut ute e non-response esponse bias)? )? April 18, 2018 www.pewresearch.org 2

Literature • Keet eter er at all ll 2000, 2006 – reduci ucing ng non-resp respons onse e in telep elepho hone ne surveys ys • Groves es and Peytche tcheva 2008 2008 – meta eta analysis alysis of studies es of absol olut ute e NR bias as • Very low correlation between response rate and non response bias • Great variation within studies then across studies • Sturg urgis s et et all l 2017 – meta eta analysi alysis focus cusing ng on relat ative e NR bias as and d FW effor ort On average, questions about beliefs and attitudes tended to respond most to FW effort, • behavioral questions not so much Small number of variables with high relative bias • • 4 call design has the most appeal • Sztabi abins nski ki 2017, Fuchs hs et et all l 2013 13 – effor ort and data a qu quali ality ty for ESS S data a April 18, 2018 www.pewresearch.org 3

Data Used • 2017 Global Attitudes Spring surveys: Nationally representative telephone and face- to-face interviews with adults 18 years and older • Scope would be 40 countries, but 10 telephone countries are included (all) • 5 f2f countries included • BUT • 15 f2f countries excluded due to response rates higher than 70% on first visit • 10 countries excluded because contact data unusable April 18, 2018 www.pewresearch.org 4

Fieldwork Efforts all F2F Countries 1 0.9 0.8 0.7 0.6 4 0.5 3 0.4 2 0.3 1 0.2 0.1 0 Source: Global Attitudes survey, 2017. www.pewresearch.org 5 April 18, 2018

Response Categories F2F Countries 100% 90% 80% 70% 60% Not eligibile 50% Non-Contact 40% Refusal 30% Interview 20% 10% 0% Source: Global Attitudes survey, 2017. www.pewresearch.org 6 April 18, 2018

CATI Countries 100% 90% 80% 70% 7.00 60% 6.00 50% 5.00 40% 4.00 30% 3.00 2.00 20% 1.00 10% 0% Source: Global Attitudes survey, 2017. www.pewresearch.org 7 April 18, 2018

CATI Countries 100% 90% 80% 70% 60% Not eligibile 50% Non-Contact 40% Refusal 30% Interview 20% 10% 0% Source: Global Attitudes survey, 2017. www.pewresearch.org 8 April 18, 2018

Percentage of Interviews Completed by Fieldwork Effort Telephone (10 countries) In-person (5 countries) Callb llbac acks Number of Number of Callb llbac acks Number of Number of complet leted intervie iews ws comple leted intervie iews ws comple leted intervie iews ws comple leted intervie iews ws 1 34% 34% 1 56% 56% 2 22% 2 29% 29% 36% 3 14% 3 14% 16% 4 10% 4 2% 17% 5 7% 6 4% 7% 7 3% Source: Global Attitudes survey, 2017. April 18, 2018 www.pewresearch.org 9

Cumulative Interview Rates – Telephone Countries 1500.00 1400.00 1300.00 Australia 1200.00 Canada 1100.00 France 1000.00 Germany 900.00 Great Britain 800.00 Japan 700.00 Netherlands 600.00 500.00 Korea 400.00 Spain 300.00 USA 200.00 Sweden 100.00 0.00 1 2 3 4 5 6 7 Source: Global Attitudes survey, 2017. April 18, 2018 www.pewresearch.org 10

Cumulative Interview Rates – F2F Countries 1200.00 1000.00 800.00 Israel Italy 600.00 Poland Hungary 400.00 Greece 200.00 .00 1 2 3 4 Source: Global Attitudes survey, 2017. April 18, 2018 www.pewresearch.org 11

Response Rates by Attempt – CATI Countries 0.40 Australia 0.35 Canada 0.30 France Germany 0.25 Great Britain 0.20 Japan Netherlands 0.15 Korea 0.10 Spain USA 0.05 Sweden 0.00 1 2 3 4 5 6 7 Source: Global Attitudes survey, 2017. April 18, 2018 www.pewresearch.org 12

Response Rates by Attempt – F2F Countries .80 .70 .60 Israel .50 Italy .40 Poland Hungary .30 Greece .20 .10 .00 1 2 3 4 Source: Global Attitudes survey, 2017. April 18, 2018 www.pewresearch.org 13

Evaluating Variable Bias Absolute bias (distance from true value – population estimate) measured after different FW effort levels measured for the following variables • Sex of respondent • Binary age variable (up to 49 years; 50 and more years of age) • Binary education variable (higher secondary education or less; above secondary education) Relative bias (distance from final estimate) measured after different FW effort levels • 4 attitudinal questions - perception and respect of US, favorability Trump & Obama • 4 technology related questions - use of internet, own cell, own smartphone, social networking Source: Global Attitudes survey, 2017. April 18, 2018 www.pewresearch.org 14

Differences by Attempt Structure – Phone (10 countries) Demographics Substantive questions A1 A1 A2 A2 A3 A3 A4 A4 T1 T1 T2 T2 T3 T3 T4 T4 Sex Sex Age Edu (50+) (4+) -1 (-7;3) 0 (-1;4) -1 (-3;1) 1 (-2;5) -1 (-4;3) 0 (-4;4) -2 (-7;1) 0 (-5;14) Easy to get (1) 0 (-6;8) 1 (-4;6) -2 (-7;2) 1 (-4;7) -1 (-12;5) Medium effort (2-3) 1 (-1;3) 0 (-4;3) -1 (-9;3) -1 (-5;2) 1 (-4;7) -1 (-8;5) 0 (-3;4) 2 (-5;8) 1 (-3;9) 0 (-6;4) 0 (-6;4) 0 (-2;5) -1 (-7;5) 2 (-2;7) 2 (-2;7) 1 (-4;8) 0 (-5;6) More effort (4-7) 0 (-11;7) -4 (-14;5) 2 (-5;8) Easy to get (1) vs. 1 (-5;14) 2 (-14;6) 2 (-2;5) -2 (-5;5) 0 (-10;7) -1 (-6;5) 2 (-4;10) -1 (-22;10) 0 (-9;6) 1 (-7;8) 4 (-5;13) Medi dium effort (2;3) Easy to get (1) vs. 1 (-8;7) 0 (-8;6) 1 (-4;7) -2 (-11;5) 2 (-3;7) 2 (-5;9) 3 (-1;14) 0 (-20;10) -1 (-19;12) -5 (-19;6) 4 (0;12) More effort (4 or more) Source: Global Attitudes survey, 2017. April 18, 2018 17 www.pewresearch.org

Differences by Attempt Structure – F2F (5 countries) Demographics Substantive questions A1 A1 A2 A2 A3 A3 A4 A4 T1 T1 T2 T2 T3 T3 T4 T4 Sex Sex Age Edu (50+) (4+) -3 (-6;-1) -1 (-3;1) Easy to get (1) 1 (-2;4) -1 (-6;3) 0 (-1;1) 0 (-2;2) -1 (-4;4) -1 (-1;3) -1 (-4;2) 0 (-3;4) -1 (-4;1) 3 (0;6) 1 (-4;9) 0 (-6;4) 4 (0;11) 0 (-6;4) 0 (-4;3) 1 (-6;4) 0 (-3;3) Medium effort (2) -3 (-6-;-1) 2 (-4;4) 1 (-4;4) -4 (-23;11) 6 (-7;22) -4 (-58;21) -3 (-9;1) -3 (-32;12) -3 (-26;5) -9 (-16;-4) 11 (6;16) More effort (3-4) -5 (-51;17) -3 (-32;22) -1 (-15;14) Easy to get (1) vs. 1 (-5;4) -4 (-8;1) 4 (-7;14) 6 (4;12) 1 (-4;12) 1 (-8;8) 6 (-1;14) 0 (9;7) -1 (-5;5) 0 (-11;8) 5 (-2;15) Medium effort (2) Easy to get (1) vs. 2 (-18;13) 4 (-7;13) -9 (-57;11) -3 (-15;11) -3 (-32;11) -3 (-26;7) -4 (-17;9) 9 (-2;20) More effort (3 or more) -4 (-51;19) -9 (-32;11) -2 (-11;4) Source: Global Attitudes survey, 2017. April 18, 2018 www.pewresearch.org 18

Demographics by Fieldwork Attempt (CATI) Low effort High effort Medium effort sex_par age_par Q001edu sex_par age_par Q001edu sex_par age_par Q001edu .15 .15 0.15 .10 0.1 .10 0.05 .05 .05 0 .00 .00 -0.05 -.05 -.05 -0.1 -.10 -.10 -0.15 -0.2 -.15 -.15 April 18, 2018 www.pewresearch.org 20

Substantive Differences by Fieldwork Effort (CATI) Low effort Medium effort A1 A2 A3 A4 T1 T2 T3 T4 A1 A2 A3 A4 T1 T2 T3 T4 .15 .15 .10 .10 .05 .05 .00 .00 -.05 -.05 -.10 -.10 -.15 -.15 Source: Global Attitudes survey, 2017. April 18, 2018 www.pewresearch.org 21

Demographics by Fieldwork Effort (F2F) Low effort High effort Medium effort sex_par age_par Q001edu sex_par age_par Q001edu sex_par age_par Q001edu 0.1 0.1 0.1 0.05 0.05 0.05 0 0 0 -0.05 -0.05 -0.05 -0.1 -0.1 -0.1 -0.15 -0.15 -0.15 Greece Hungary Israel Italy Poland Greece Hungary Israel Italy Poland Greece Hungary Israel Italy Poland April 18, 2018 www.pewresearch.org 23

Substantive Differences by Fieldwork Effort (F2F) Low effort Medium effort A1 A2 A3 A4 T1 T2 T3 T4 A1 A2 A3 A4 T1 T2 T3 T4 0.12 0.12 0.1 0.1 0.08 0.08 0.06 0.06 0.04 0.04 0.02 0.02 0 0 -0.02 -0.02 -0.04 -0.04 -0.06 -0.06 -0.08 -0.08 Greece Hungary Israel Italy Poland Greece Hungary Israel Italy Poland Source: Global Attitudes survey, 2017. April 18, 2018 www.pewresearch.org 24