National Center for Health Statistics Does Reducing the Length Matter? An Assessment of Respondent Burden with the Redesigned U.S. National Health Interview Survey James Dahlhamer, Aaron Maitland, Ben Zablotsky, Adena Galinsky National Center for Health Statistics, USA DC-AAPOR Workshop on Respondent Burden This presentation does not represent official opinions or statistics of the National Center for Health Statistics or the Centers for Disease Control and Prevention
U.S. National Health Interview Survey (NHIS) Purpose : To monitor the health of the U.S. population through the collection and analysis of data on a broad range of health topics Sample : Address-based, multi-stage, clustered national sample of housing units from every state, to be representative of the civilian noninstitutionalized US population Mode : In-person interviews by U.S. Census Bureau interviewers, with follow- up by telephone if needed Data collection : Continuous, with quarterly and annual data files Target sample size : Complete interviews for 27,000+ households Content redesign : January 2019
2019 NHIS Questionnaire Redesign Goals of the redesign – Improve the relevancy of covered health topics, better meeting the needs of data users – Focus on leading causes of morbidity and mortality, targets of health promotion initiatives, and risk and protective factors – Harmonize overlapping content with other federal heath surveys – Establish a long-term structure of ongoing and periodic topics – Reduce respondent burden
2019 NHIS Questionnaire Redesign: Reducing Burden Time-related burden – Eliminated topics better collected by other federal surveys – Eliminated questions on rare conditions – Limited the length of sponsored content Cognitive and emotional burden – Improved clarity of questions – Grouped similar content together to improve flow – Moved health questions to beginning, sensitive questions to end Design changes affecting both – Household roster → Family interview → Sample child → Sample adult
Research Questions Per respondent assessments, did we reduce respondent burden with the questionnaire redesign? Do respondent assessments of the ease or difficulty of answering and the sensitivity of questions differ between the old and new questionnaires? What impacts do they have on respondent assessments of burden? To what extent did we reduce the length of an NHIS interview? What impact does time spent in the interview have on respondent assessments of burden?
Burden Questions Main burden question: – How burdensome was this survey to you? Would you say not at all burdensome, a little burdensome, moderately burdensome, very burdensome, or extremely burdensome? Ease or difficulty of answering: – How easy or difficult was it for you to answer the questions in this survey? Would you say very easy, somewhat easy, somewhat difficult, or very difficult? Sensitivity of questions: – How sensitive were the questions in this survey? Would you say not at all sensitive, a little sensitive, moderately sensitive, very sensitive, or extremely sensitive?
Data/Methods (1) Households randomly assigned to receive either the old or new questionnaire in quarter 4 (October, November, December) of 2018 Burden questions asked at the end of the sample adult or sample child module in fully complete interviews Bivariate analyses of the following by questionnaire: – Respondent assessment of burden – Respondent assessment of ease/difficulty of answering – Respondent assessment of the sensitivity of questions – Time spent in the interview
"How Burdensome Was This Survey to You?" by Questionnaire: NHIS, 2018, Quarter 4 (base weights) 2.0% 1.1% 100% 2.3% 3.1% 8.3% 90% 12.1% 80% 24.8% 70% 29.6% 60% Χ 2 p-value < .01 50% 40% 63.5% 30% 53.1% 20% 10% 0% Old Questionnaire New Questionnaire Not at all A little Moderately Very Extremely
"How Easy or Difficult Was it for You to Answer the Questions in This Survey?" by Questionnaire: NHIS, 2018, Quarter 4 (base weights) 0.7% 0.5% 100% 3.3% 5.2% 90% 19.1% 25.4% 80% 70% Χ 2 p-value < .01 60% 50% 40% 77.0% 68.9% 30% 20% 10% 0% Old Questionnaire New Questionnaire Very easy Somewhat easy Somewhat difficult Very difficult
"How Sensitive Were the Questions in This Survey?“ by Questionnaire: NHIS, 2018, Quarter 4 (base weights) 1.5% 1.2% 100% 3.8% 4.0% 90% 12.4% 13.1% 80% 70% 32.2% 34.3% Χ 2 p-value = .112 60% 50% 40% 30% 50.5% 47.2% 20% 10% 0% Old Questionnaire New Questionnaire Not at all A little Moderately Very Extremely
Time Sample Adult Spent in the Interview by Questionnaire: NHIS, 2018, Quarter 4 (base weights) 100% 4.7% 7.4% 18.6% 90% 80% 20.3% 17.4% 70% Χ 2 p-value < .01 60% 25.6% 50% 40.8% 40% 30% 27.9% 20% 26.8% 10% 10.5% 0% Old Questionnaire New Questionnaire < 30 minutes 30 - < 45 minutes 45 - < 60 minutes 60 - < 75 minutes 75+ minutes
Data/Methods (2) Model respondent assessments of burden using logistic regression – Dependent variable = moderately/very/extremely burdensome (high burden) vs. a little/not at all burdensome – Independent variable: questionnaire (new vs. old) – Covariates: perceptions of survey content, task difficulty, recruitment effort and respondent motivation, and sample adult and case characteristics Model was estimated separately for the old questionnaire (n=3,943) and new questionnaire (n=3,940), and then applied to the full dataset (n=7,883) with an indicator for questionnaire
Covariates Included in Logistic Regressions of Respondent Burden (1) Perceptions of Survey Content – Ease/difficulty of answering questions – Sensitivity of questions Measures of Task Difficulty – Total minutes sample adult spent in interview
Covariates Included in Logistic Regressions of Respondent Burden (2) Measures of Recruitment Effort and Motivation – Householder(s) expressed time constraints – Householder(s) made refusal-like statements – Householder(s) expressed privacy concerns – Proportion of sample adult responses don’t know or refused – Number of contact attempts – Case reassigned to different interviewer
Covariates Included in Logistic Regressions of Respondent Burden (3) Sample Adult and Case Characteristics – Age – Sex – Education – Race/ethnicity and language of interview – Reported health status – Cognitive difficulty – Number of children – Mode of interview (FTF vs. phone)
Significant (p < .05) Adjusted Odds Ratios from Logistic Regressions of Respondent Burden: Questionnaire, Perception of Survey Task, and Task Difficulty Measures Combined Old New New questionnaire (vs. old) 0.55 --- --- Somewhat easy/somewhat difficult/very 2.42 2.51 2.28 difficult (vs. very easy) Sensitivity of questions (ref = not at all) A little 1.56 1.78 1.33 Moderately/very/extremely 5.76 6.29 5.30 Total sample adult minutes in interview (ref = > 45 -- < 60) < 30 minutes > 30 -- < 45 minutes > 60 -- < 75 minutes > 75 minutes 1.31 1.34
Summary of Remaining Significant Effects from Logistic Regressions of Respondent Burden Combined Old New Householder(s) expressed time constraints * * * Householder(s) made refusal-like statements * * Householder(s) expressed privacy concerns * * * Number of contact attempts * Sample adult item nonresponse * * * Mode of interview * * Male * Age * * * Education * Race/ethnicity and language * * Number of children * * * Case reassigned to a different interviewer, cognitive difficulties, and reported health status were not associated with burden in any of the models.
Conclusions (1) Sample adults who received the new questionnaire reported lower levels of burden compared with sample adults who received the old questionnaire Items in the new questionnaire appear to be easier to answer, but perceptions of sensitivity did not differ between the surveys Time was significantly reduced in the new instrument – Time spent in the interview was associated with burden in the old, but NOT the new questionnaire
Conclusions (2) Respondent assessments of ease/difficulty of answering and sensitivity of questions were the strongest predictors of burden – Design changes to reduce cognitive and emotional burden appear to be driving the reductions in burden observed with the new questionnaire
Thank you! jdahlhamer@cdc.gov
Extra Slides
"How Burdensome Was This Survey to You?" by Questionnaire: NHIS, 2018, Quarter 4 (base weights) 2.0% 1.1% 100% 2.3% 3.1% 11.7%* 8.3% 17.2%* 90% 12.1% 80% 24.8% 70% 29.6% 60% 50% 40% 63.5% 30% 53.1% 20% 10% 0% Old Questionnaire New Questionnaire Not at all A little Moderately Very Extremely *p < .05
NHIS Response Rate and Completed Interview Times 100% 150 88% Response Rate 120 75% 67% 90 90 Minutes 66 50% 60 Mean Interview Time 25% 30 Response Rate 0% 0 2001 2003 2005 2007 2009 2011 2013 2015 2017
Length of Completed Interviews in Minutes: NHIS, 2018, Q4 Old New Questionnaire Questionnaire Mean 77.4 55.1 Median 70.4 48.9
Recommend
More recommend
