Nonresponse Bias J. Michael Brick, Westat Roger Tourangeau, Westat - PowerPoint PPT Presentation

Responsive Designs to Reduce Nonresponse Bias J. Michael Brick, Westat Roger Tourangeau, Westat Adaptive Survey Design Workshop March 14, 2018

Premise • Increasing nonresponse rates lead to increased chance of nonresponse bias in estimates and increased data collection costs • Responsive/adaptive design is a tool to help conduct surveys efficiently in this environment while lowering nonresponse bias 2

Premise • Increasing nonresponse rates lead to increased chance of nonresponse bias in estimates and increased data collection costs • Responsive/adaptive design is a tool to help conduct surveys efficiently in this environment while lowering nonresponse bias • Requires – understanding relationship between nonresponse rates and bias AND of field operations available to reduce nonresponse bias 3

Outline • Re-examine relationship between nonresponse rate and nonresponse bias • Discuss why the nonresponse rate to bias relationship is not stronger • Implications for Responsive Design • Conclusions 4

Nonresponse Rates and Bias • The theory    f Cov ( , y ) f f   y , y i i ( ) NR Bias y f f r  f   cv ( ) f y , y • Both the mean and standard deviation of the propensities ( f ) are important. • E.g., the cv( f ) =0.5 when f =.80 and cv( f ) = 2.0 (4 times larger) when f =.20….huge increase in f needed to keep bias the same 5

Empirical Studies • Contrary to expectations, widely cited studies show little or no relationship — Curtin, Presser, and Singer (2000) — Groves (2006) — Groves and Peytcheva (2008) — Keeter, Miller, Kohut, Groves, and Presser (2000) — Keeter, Kennedy, Dimock, Best, and Craighill (2006) — Merkle and Edelman (2002) 6

Groves and Petcheva (2008) Meta-Analysis • Most comprehensive and influential study • Two conclusions — Little or no relationship between bias and rate — Tremendous within-study variability in bias • Second conclusion suggests no study-level indicator is informative about nonresponse bias • G&P provided their data set for our re-analysis: 959 relbias estimates from 59 studies 7

Groves and Peytcheva (2008) • Looked at 59 studies with bias estimates (959 estimates) 8

Reanalysis — G&P Data, by Sample Size 40 35 30 25 Mean Absolute Relbias 20 15 10 5 0 15 25 35 45 55 65 75 85 95 -5 Study Response Rate 9

Correlation between Response Rates and RelBias Measures at the Estimate and Study Level Correlation Unweighted Estimate-Level Correlations Response rate and absolute relbias -.191 (n=953) Unweighted Study-Level Correlations Response rate and mean absolute relbias -.255 (n=57) Study-Level Correlations — Weighted by Number of Estimates Response rate and mean absolute relbias -.402 (n=57) Study-Level Correlations — Weighted by Study Sample Size Response rate and mean absolute relbias -.413 (n=57) 10

Reanalysis Conclusions • There is a relationship between nonresponse rate and bias at the study-level • Some additional study-level characteristics beside nonresponse rate are important (e.g., method of estimating bias) • Big differences in bias by study; study accounts for much of the variance, about a quarter 11

Four Models for Relationship (1) • Random propensities : Propensities essentially random, product of many transient characteristics of respondents — “… We’re sort of lucky. The mechanisms that produce the decision to participate or not participate in a survey are myriad; …the covariance between the decision to participate and what we’re measuring tends to be small..” (Groves, 2017) • Design-driven propensities : Response propensities largely determined by study-level design features that are largely unrelated to characteristics of the sample members 12

Four Models for Relationship (2) • Demographic-driven propensities: Propensities determined by respondent characteristics unrelated to survey variables or corrected by weighting — Low bias • Correlated propensities: Response propensities determined by design features and characteristics of the sample members; some groups (high education, voters, civically engaged, altruistic) consistently respond at high rates — High bias 13

Large Biases • A set of variables related to a sense of civic obligation and volunteering are highly related both to survey participation and these variables. • Survey estimates involving these variables such as reports about voting are at substantial risk for large biases (Tourangeau, Groves, & Redline, 2010) Bias Entire Sample Respondents Subgroup (Frame Data) (Frame Data) Nonresponse Overall 43.7 (2689) 57.0 (904) 13.3 Telephone 43.2 (1020) 57.4 (350) 14.2 Mail 43.9 (1669) 56.7 (554) 12.8 14

Theory and Empirical Results • Substantial biases generally rare, although rel-biases may be large • Large biases often appear for a specific set of variables that are correlated with unit response (e.g., volunteering) • Weighting using correlates like education help reduce bias somewhat • For most estimates, random propensities and design-driven propensities seem reasonable models 15

Responsive/Adaptive Design • Responsive designs: Designs with multiple phases, with aim of reducing bias by getting more representative set of respondents (Groves and Heeringa 2006) • Adaptive designs: Designs tailored from the outset (Luiten and Schouten, 2013) or adapted continuously throughout the field period (Peytchev, Riley, Rosen, Murphy, and Lindblad 2010) to achieve more balanced set of respondents 16

Implications • Average study-level nonresponse bias can be reduced by lowering overall nonresponse rate • Large decrease in nonresponse rate needed to achieve reductions in bias • Targeting data collection efforts to reduce variation in response propensities may be more effective strategy • Since large nonresponse bias generally associated with specific variables, any efforts to target those cases my be most promising strategy 17

Data Collection Tools • Avoid designs that give equal effort to all cases • Adaptive/responsive designs employ unequal efforts — Change/vary modes — Change/vary incentive levels — Vary level of effort for different cases or subgroups — Two-phase sampling and focus effort • Auxiliary data related to propensities (e.g., volunteering) are extremely important but often unavailable 18

Conclusions • There is a relationship between nonresponse rates and bias, but it is not strong • Models suggest that many estimates are likely to have small biases if data collection efforts are reasonable • Targeting efforts to reduce variation in propensities is worthwhile if possible • Variables with worst biases may be hard to affect because auxiliaries unavailable in most cases • Weighting helps too 19

THANK YOU!!!! mikebrick@westat.com 20