Interview Methodology for use with Spanish-Speaking Respondents - PowerPoint PPT Presentation

Adaptation of Standard Cognitive Interview Methodology for use with Spanish-Speaking Respondents Patricia Goerman and Ryan King, U.S. Census Bureau Presented at the 69 th annual conference of the American Association for Public Opinion Research (AAPOR) Anaheim, CA: May 15-18, 2014

Outline of talk  Cognitive interviewing (CI)  Past research  Research questions  Study methods  Findings  Next steps

Cognitive interviewing (CI) Definition:  One-on-one interviews to evaluate whether respondents interpret, comprehend and respond to survey questions as intended. Think aloud procedure Probes (concurrent or retrospective)

Cognitive interview probes Meaning oriented  “ What does the term ‘foster child’ mean to you in this question?” Process oriented  “How did you arrive at/choose that answer?” Paraphrasing  “ Can you tell me in your own words what that question is asking?” Recall  “How do you remember that you (saw a dentist 3 times) in the last year?”

Past research  Monolingual cognitive interview (CI) methods:  Presser et. al. 2004; Willis 2005; Beatty and Willis 2007; Blair and Conrad 2011  Adaptation of CI Method for use across languages/cultures:  Dean et al. 2007; Willis and Zahnd 2007; Fitzgerald and Miller 2009; Pan et al. 2010; Harkness et al 2010

Research questions  Why have previous studies found that CI techniques cause discomfort among non- English respondents?  What specific interview techniques and probes work best with Spanish-speaking CI respondents?

Methods  48 cognitive interviews  Spanish speakers  Segment of American Community Survey (ACS): CATI/CAPI  Types of question tested:  Person level: age, sex, DOB, marital, Hispanic origin/race, place of birth, year of entry, education, ancestry, language spoken, English proficiency  Household level: Type unit, year built, when moved, number of acres, number rooms and bedrooms, plumbing, telephone, vehicles

Demographic characteristics of Spanish speakers (n=48) National origin Gender 44 Mexican origin 42 Female 4 Some other origin 6 Male Education level Age 35 Less than HS Range: 24-74 13 HS diploma or more Mean: 46 Median: 44 Missing: 4

Type of interviews: Standard v. experimental  Variation of introduction and probe wording  Standard interviews  Direct translation of typical U.S. English protocol (procedure and wording)  Experimental interviews  Variation of introductory statements/conversation  Different sample probes to begin  Flexibility in probe wording

Pre-interview interactions: Both standard and experimental  Discussions of the interview process prior to beginning  Reason for doing the interview  Confidentiality  Reasons for taping the interview

Pre-interview interactions: Experimental v. standard Experimental Standard N=28 N=20 Information provided about Int. 36% 10% Family Background 36% 5% How learned Spanish 36% 5% Time spent abroad 11% 5% Discuss Job at Census 36% 0% Small talk between interviewer and respondent about: The respondent 71% 40% Interview location 61% 60% Census Bureau / surveys 39% 10% Interview city 18% 0%

Coding scheme/analysis Yes/no codes: 1. Was probe administered? 2. Was probe administered as worded in protocol? 3. Did respondent understand when read as worded? 4. Was probe reworded? 5. Was probe understood when reworded? 6. Did probe cause discomfort? 7. Did respondent provide “useful” answer to probe? Descriptive codes: 1. Type probe rewording done? (description) 2. What type of discomfort did it cause? (description)

Inter-coder reliability  Application of Kappa statistic  Kappa Scores  Total of 7 cases coded by both interviewer/coders  Overall: 0.68 (n=892)  Yes/No: 0.71 (n=784) (good)  Type probe rewording: 0.50 (n=69) (fair-good)  Discomfort: 0.38 (n=39) (poor)

Frequency of rewording by probe type and interview type 100% 90% 86% 90% 85% 77% 76% 80% 70% 61% 60% 50% 41% 40% Experimental 30% 26% 30% Standard 21% 20% 10% 0% Meaning Paraphrasing Process Think Recall Oriented E=92, Oriented Aloud E=17, E=132, S=94 E=44, E=48, S=19 S=94 S=61 S=47

Usefulness of probes by interview type 100% 100% 91% 89% 90% 77% 80% 75% 74% 72% 70% 70% 60% 50% 43% 42% 40% Experimental 30% Standard 20% 10% 0% Meaning Paraphrasing Process Think Recall Oriented E=92, Oriented Aloud E=17, E=132, S=94 E=44, E=48, S=19 S=94 S=61 S=47

Overall Discomfort 100% 90% 80% 70% 60% 53% 50% Experimental 40% 31% Standard 30% 21% 21% 18% 17% 20% 16% 15% 13% 12% 8% 6% 10% 2% 0% 0% Total Pause Embarass Annoying Emotional Laughing Other E=333 E=27 E=8 E=3 E=0 E=9 E=4 S=315 S=11 S=16 S=6 S=1 S=7 S=11

Overall usefulness of probes by education level and treatment 100% 90% 80% 70% Less Than High School - 60% Experimental 50% Less Than High School - 40% Standard 30% More Than High School - 20% Experimental 10% More Than High School - 0% Standard Meaning Paraphrasing Process Think Recall Oriented LTHS-E=69, Oriented Aloud LTHS-E=14, LTHS-E=99, LTHS-S=64, LTHS-E=33, LTHS-E=33, LTHS-S=14, LTHS-S=68, MTHS-E=23, LTHS-S=40, LTHS-S=32, MTHS-E=3, MTHS-E=33, MTHS-S=30 MTHS-E=11, MTHS-E=15, MTHS-S=5 MTHS-S=26 MTHS-S=21 MTHS-S=15

Frequency of “successful” major probe rewording 100% 100% 100% 100% 91% 90% 82% 80% 79% 80% 70% 59% 60% 50% 50% 40% 35% Experimental 30% Standard 20% 10% 0% Meaning Paraphrasing Process Think Recall Oriented E=14, Oriented Aloud E=12, E=95, S=2 E=22, E=36, S=3 S=20 S=17 S=31

Probe rewording: Think aloud  Original/scripted probes:  ¿Podría pensar en voz alta cuando está decidiendo cómo contestar la pregunta?  Por favor dígame qué está pensando.  Rewording:  Y si quisiera hablar mientras piensa, me gustaría .  Me encantaría oír lo que está pensando mientras mira…

Probe rewording: Paraphrase  Original/scripted probe:  ¿Podría decirme con sus propias palabras qué información pide esa pregunta?  Rewording:  ¿Y qué información le esta pidiendo esta pregunta? ¿Qué le parece?  Si ud tuviera que preguntar eso a alguien, cómo lo diría?  ¿Qué le parece que quieren preguntar (o saber)?

Next steps  Additional analysis:  Look at emergent probes in experimental cases  Look at debriefing, people’s experience with reading, literacy, education, etc.  New research:  Different national origin Spanish speakers  Speakers of additional languages

Adaptation of Standard Cognitive Interview Methodology for use with Spanish-Speaking Respondents Patricia Goerman and Ryan King, U.S. Census Bureau For more information: E-mail: Patricia.L.Goerman@census.gov Disclaimer: This presentation is intended to inform people about research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.