During and after fieldwork: Fieldwork monitoring, quality control - PowerPoint PPT Presentation

During and after fieldwork: Fieldwork monitoring, quality control and several ways to assess nonresponse bias Joost Kappelhof

Overview  Paradata & monitoring fieldwork  Nonresponse and nonresponse bias  Ways to assess nonresponse bias  Several ways to deal with nonresponse  Alternative quality indicators and nonresponse bias INGRID 2016

Paradata  Couper (1998) and can mean process data (e.g. call records, timers, etc.) and/ or auxiliary information, such as sample frame data, interviewer observations, etc. • Various sources, such as sample frames, contact form s , timers, etc.  Useful for: • Find out if fieldwork has been carried out according to specifications • Number of calls, Timing of calls • Monitoring fieldwork • When and how to intervene, interviewer performances, quality control • Calculate response rates • Detect fraud • Nonresponse analysis, bias adjustment and future recommendations • Number of calls, Timing of calls, Reasons for refusal, Compare early and late contacts, Compare easy and difficult respondents, final disposition codes. INGRID 2016

Example of how and what type of paradata is collected in the ESS  Contact form s in the ESS • Form to be completed, coded and keyed for every sample unit • Different versions for address, household and individual sample frames • For address samples: household selection procedure (if 1+ ) • For address and household samples: person selection procedure (if 1+ ): Kish grid or first/ last birthday • Detailed call records (to be coded and keyed) • Lots of other information (characteristics dwelling, etc.) • Different fieldwork procedures (phone calls allowed in Nordic countries) INGRID 2016

Contact form: for each sample unit  interviewer number  target person name and telephone number  selection of households for address samples  selection of persons for household samples  Interviewer judgment on future cooperation (in case of refusal)  neighbourhood characteristic form • type of dwelling • physical state of building and neighbourhood • presence of litter/ rubbish • Vandalism/ graffiti in neighbourhood). INGRID 2016

Call records  for each call • date, exact time, mode • result (complete/ partial interview, different types of noncontact, ineligible)  for each contact • Final outcome (interview, appointment, refusal (by target person or someone else) • different types of not being able to participate (language barrier)  if ineligible • why invalid address  for each refusal • reasons for refusal and judgments of possible cooperation at future calls • sex and age of doorstep contact INGRID 2016

Problems and challenges with the ESS CF and CF data  Different privacy regulations • Privacy: no reasons for nonresponse • Privacy: no observational data • Privacy: no information on nonrespondents at all  Forms complicated  Errors in some countries (final interview before refusal?)  Codes were missing or unclear • how do we know if someone has moved out of the country?  Telephone calls do not fit (hard to keep or too many)  Data sometimes too perfect  Comparability of interviewer observations.  Self-fulfilling prophecy with interviewer judgment future cooperation  How far can we go? INGRID 2016

Nonresponse  Types of nonresponse  Unit and item  Types of missingness (Rubin, 1976)  MCAR (Missing Completely At Random)  E.g., The reason for the missing data occurs entirely at random. It is independent of observable variables, such as gender, and the unobservable parameter of interest, such as income  MAR (Missing At Random -given the observed data, the missingness mechanism does not depend on the unobserved data )  E.g., Men are less likely to answer a question about income, but the missingness has nothing to do with income level after you adjust for gender  MNAR (Missing Not At Random – nonignorable nonresponse)  E.g., The reason an answer on income is missing is correlated in an unknown way with the level of the income (and not on observed variables such as gender) INGRID 2016

From nonresponse to assessing nonresponse bias  (Unit) Nonresponse happens at the survey level  Nonresponse bias is a characteristic of a variable, not of a survey  How do you know there is nonresponse bias?  Can you measure nonresponse bias?  Can you adjust for nonresponse bias? INGRID 2016

INGRID 2016

Response models Fixed Response Model N 1 ∑  The estimator: = y a R Y R k k k n = R k 1  The expected value: = E(y ) Y R R N  The bias: = − = − = NR B(y ) Y Y (Y Y ) QK R R R NR N  The bias of the estimator is determined by: • Average difference between respondents and nonrespondents. • Relative size of the nonresponse stratum. The Nonresponse Problem (Jelke Bethlehem, selection)

Response models Random Response Model N 1 ∑  Estimator: = y a R Y R k k k n = R k 1 ρ N Y 1 ∑  Expected value: ≈ ≡ * k k E(y ) Y , r ρ N = k 1 ρ where is the mean response probability. R S S ρ ρ  Bias of the estimator: = − = Y Y B(y ) E(y ) Y ρ R R The bias is large if  The target variable and response behaviour are correlated.  There is much variation in response probabilities.  The response rate is small. The Nonresponse Problem (Jelke Bethlehem, selection)

When to expect nonresponse bias?  More men than women?  More poor than rich?  More employed than unemployed?  Related to topic of survey • Time use • Employment • Political interest • Volunteer work • Travel • Interest in topic INGRID 2016

5 approaches to assess nonresponse bias Groves, 2006 1. Compare response rates across subgroups 2. Comparisons to similar estimates from other sources 3. Look at variance within existing survey 4. Using rich sampling frame data or suppl. matched data 5. Contrasting alternative postsurvey adjustments for nonresponse INGRID 2016

1. Compare response rates across subgroups  Presenting estimates of response rates on key subgroups of the target population (e.g., age, race, gender, urbanicity subgroups). Generally, the researcher asserts that there is no evidence of “nonresponse bias” if the response rates are similar across subgroups.  Easy, but can be misleading (not informative about possible nonresponse biases in estimates based on other survey variables)  Peytcheva, E., Groves, R.M. (2009). Using variation in response rates of demographic subgroups as evidence of nonresponse bias in survey estimates. Journal of Official Statistics, 25, 193 – 201. INGRID 2016

2. Comparisons to similar estimates from other sources  For example, to compare the distributions of age, gender, and other sociodemographic variables among respondents with those from gold standard (high-quality) survey or population statistics  Pro: • Estimates independent of the survey are compared • Credibility of reference survey can give confidence about estimates of the survey in question  Con: • Key survey variables of the survey do not usually exist in the external source • Differences in measurement between survey and gold standard • Coverage and nonresponse characteristics of the gold standard survey are not completely known INGRID 2016

3. Variance within existing survey  For example, compare early and late respondents (differences in contactability) or cooperative and reluctant respondent.  Pro: • Can be used in many different modes of data collection, with diverse populations, on diverse topics.  Con:  Offers no direct information about the nonrespondents to the survey (assumption of continuum of resistance)  Requires process data and the construction of indicators on contactability and reluctance can be tricky INGRID 2016

How to measure contactability INGRID 2016

Indicators of contactability Number of calls to first contact Number of calls INGRID 2016

Contactability First Call (nr, time, mode) contact INGRID 2016

Problems when measuring contactability  Has the target respondent been contacted?  Acknowledge timing and mode of calls  What is optimised? • Minimum number of contacts • Costs of contacts  Individual interviewer strategies • Adapt calling pattern to at-home behaviour respondents (from start or after first call) • How are interviewers paid  Rarely at home or away for longer period INGRID 2016

How to measure reluctance INGRID 2016

Indicators of reluctance  Sampling frame • Comparing reluctance across countries  One contact • Duration of recruitment • Doorstep interaction comments Couper, 1997; Campanelli, Sturgis and Purdon, 1997; Loosveldt and Storms, 2001 • Interviewer judgment (How difficult was it … ?)  More contacts: Soft and hard refusals • Number of contacts (1, 2+ ) • Reason for initial refusal (topic, survey, unspecific) • Interviewer judgment: likelihood of future cooperation • Extra incentives INGRID 2016