Assessing the Burden of the American Community Survey Jessica Holzberg Center for Behavioral Science Methods DC-AAPOR Respondent Burden Workshop October 21, 2019 Disclaimer: This presentation is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau. CBDRB-FY19-CED001-B0018
Burden is “the product of an interaction between the nature of the task [objective burden] and the way it is perceived by the respondent [subjective burden ]” (Bradburn 1978, p. 36) 2
What might contribute to burden in a household survey? Survey Respondent characteristics characteristics • • Questions Demographics Respondent • Length (age, education, perceptions • Response mode etc.) • • Contacts (#, Attitudes • mode, message) Interest • • Mandatory Motivation • • One-time or Household size Respondent longitudinal burden Adapted from Fricker (2016) 3
The American Community Survey (ACS) • The ACS produces estimates on more than 35 topics: • Demographic (age, sex, race, relationship to householder) • Social (marital status, disability, school enrollment, veteran status) • Economic (health insurance coverage, employment status, income/earnings) • Housing (tenure, year structure built, home heating fuel) • The ACS takes an average of 40 minutes to complete • Response to the survey is mandatory 4
The ACS (cont.) • ACS samples approximately 3.5 million addresses annually (about 290,000 addresses per month) • Response modes • Self-response • Internet • Paper • Interviewer-administered response (if no self-response received) • CATI (until October 2017) • CAPI 5
Present study • Until recently, ACS research has focused on objective burden • Goal: Develop questions to measure respondents’ subjective perceptions of burden of the ACS • Literature review • Focus groups • Cognitive interviews • Questions may be field tested and implemented on the ACS as optional, follow-on questions 6
Research questions 1. What is respondents’ overall level of perceived burden in being contacted by and responding to the ACS? 2. Which features of the ACS contribute to or affect respondents’ level of perceived burden? How much does each of these survey features contribute to respondents’ perceived burden ? 3. Which respondent and household-level characteristics, if any, contribute to variation in respondents’ perceived level of burden? 7
Focus groups (spring 2017) • 10 focus groups with 46 ACS respondents held across 3 cities • 5 self-response groups, 5 interviewer-administered response groups • 3-8 respondents* per group • Respondents recruited by telephone • Recruitment targets focused on response mode; demographics collected during focus groups • Fairly even on sex, race (white v. nonwhite), education (less than Bachelor’s degree v. Bachelor’s degree or higher), age (younger than v. older than 45) • Most were non-Hispanic and lived in 1-2 person households *Variation in size of the focus groups was by design as part of separate methodological research 8
Focus groups (cont.) • Moderator’s guide 1. Open discussion of ACS experiences 2. 13 survey questions on the burden of the ACS, mostly adapted from 2013 Consumer Expenditure Survey 9
Takeaways from focus groups • Respondents did not consider the ACS to be burdensome • “Took a little time for me to do it, but I wouldn’t have exactly called it burdensome.” • Diversity in how respondent perceptions were formed • “It only took about 10, 15 minutes. He asked me basic questions…” • “It had a lot of questions.” 10
Takeaways from focus groups (cont.) • Most respondents were thinking about features of the questionnaire when considering the burden of the survey • Length of the survey, question difficulty, question sensitivity • Fewer comments on other aspects of survey, such as contacts • “It was just the right amount for me because I kept forgetting.” • Some evidence of potential differences in perceptions by respondent characteristics • “I think a lot depended [on] if you had 3 people in your household versus 7 in your household.” • “I just felt like it wasn’t my information to give, I guess.” 11
Questions for cognitive testing • Overall burden of the ACS (two versions: “burdensome” v. “hassle”) • Questionnaire • Length of survey response • Whether length of survey response met respondents ’ expectations • Difficulty answering ACS questions • Contacts • Number of contact attempts • Appropriateness of contacts • Level of effort put forth in responding to the ACS • Perceptions of whether the ACS “invaded” their privacy 12
Cognitive interviews (fall 2017-summer 2018) • 62 cognitive interviews across 2 rounds and 5 cities • Prior ACS respondents and newly recruited respondents • New respondents took ACS prior to cognitive interview • Recruitment • Prioritized diversity in demographics, household composition, response mode • Prior ACS respondents recruited as in focus groups • Traditional cognitive interview recruitment methods used for new respondents (Craigslist, flyers, etc.) 13
Cognitive interviews (cont.) • Demographics • Fairly diverse on sex, race (white v. nonwhite), education (less than Bachelor’s degree v. Bachelor’s degree or higher), age (younger than v. older than 45), household size, owning v. renting • Most were non-Hispanic, income $50,000+, and lived only with related household members • Respondents answered burden question series and were debriefed on their answers • Question revisions made between rounds 14
Takeaways from cognitive interviews • Similar to the focus groups • Respondents did not consider the ACS to be burdensome • “It didn’t bother me at all to do it.” • No big differences in comprehension, response to two versions of the overall burden question • “Hassle” more plain language but also perhaps too informal 15
Takeaways from cognitive interviews (cont.) • Most respondents were thinking about the questionnaire when considering the overall burden of the survey • “It really did not take that much time to complete.” • “The survey itself was pretty straightforward.” • Some evidence of potential differences by response mode, household composition and size • “I don’t know if it’s unique to my situation or not but there were a lot of questions I couldn’t answer…” 16
Selecting questions for further testing • Questions generally well understood and able to be answered • Asking questions about burden can increase burden of a survey • Need to be selective, implement only a minimal number • Priority given to questions that captured overall perceptions or unique perceptions that other questions did not 17
Evolution of burden questions across cognitive interviews Round 1 Round 2 Recommendations for field testing Overall burden of the ACS Overall burden of the ACS Overall burden of the ACS • • • “Burdensome” “Burdensome” “Burdensome” • • • “Hassle” “Hassle” “Hassle” Questionnaire: Questionnaire: Questionnaire: • • • Response length Response length Response length • • Perception of length Perception of length • • Difficulty Difficulty Contacts Contacts • • Number of contact attempts Number of contact attempts • • Perception of contacts Perception of contacts Effort Effort Effort Whether ACS “invades” privacy Whether ACS “invades” privacy Whether ACS “invades” privacy (The exact wording of retained questions was revised after round 1 and at the conclusion of cognitive testing) 18
Limitations • Gap between ACS response and focus group/cognitive interview participation • Unlikely to have recruited the most burdened respondents 19
Conclusion • Burden is a complex, multidimensional construct • Generally, clear need for more research on measuring subjective perceptions of survey burden • Question wording: Difficult to come up with a broad term that captures “burden”! • Differences by survey and respondent characteristics • Final report on ACS burden testing forthcoming 20
Official call coming soon: Journal of Official Statistics Special Issue on Respondent Burden 1. Quantitative research on the relationship between respondent burden, response propensity, nonresponse bias, response rates, item nonresponse, and other data quality outcomes 2. Qualitative research on respondents’ subjective perceptions of survey burden 3. Differences in respondent burden across different survey modes 4. The effects of objective versus subjective respondent burden and the relationship between them 5. The effects of survey design on respondent burden 6. Techniques for reducing respondent burden 21
Jessica.Holzberg@census.gov 22
Recommend
More recommend
