Integrating Dependent Interviewing into a CAPI EHC: Challenges and Opportunities for the Survey of Income and Program Participation – Event History Calendar (SIPP-EHC) Jason Fields Brianne Hillmer US Census Bureau September 22, 2011 Methodology Seminar University of Michigan – Institute for Social Research Dependent data - 2012 SIPP-EHC 1
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS Outline • Context • SIPP History and Design • Panel Surveys – Dependent Data & Seams • SIPP-EHC • Constraints and Cautions • Evaluation Plans Dependent data - 2012 SIPP-EHC 2
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS Context • The Survey of Income and Program Participation - SIPP • Sponsor / coordinating subject matter division • Multiple Census Bureau areas involved • Significant external stakeholder interest • This is work in progress Dependent data - 2012 SIPP-EHC 3
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS The Save the SIPP Campaign New York Times Editorials Discovering What Happens Next Save our SIPP The Continuing Saga of SIPP NY Representative, Carolyn Maloney stated: “I want to thank all of the policy groups and economists who worked so hard to help reverse the Administration’s original SIPP decision. Because of their dedication and hard work, the Administration came to understand how important SIPP is to creating and implementing good public policy.” Dependent data - 2012 SIPP-EHC 4
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS SIPP-EHC Implement Improvements to SIPP - Reduce costs - Reduce respondent burden - Improve processing system - Modernize instrument - Expand/enhance use of administrative records Key Design Change: - Annual interview, 12-month reference period, Event History Calendar (EHC) methods Dependent data - 2012 SIPP-EHC 5
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS Timeline from SIPP to SIPP-EHC • December 2005 • Conversion of the SIPP 2004 Panel Instrument to BLAISE and reprogramming of the post-data collection processing into SAS stopped. • January 2006 • First meeting with key SIPP stakeholders to discuss future plans for SIPP. • March – May 2006 • “Save the SIPP” letter was circulated in reaction to the potential loss of SIPP and the alternatives suggested. • June – December 2006 • Several stakeholder meetings held and the basic structure of a new SIPP survey discussed • June – July 2007 • Senate and House Appropriations propose funding to continue SIPP collection in 2008 and develop a more cost effective alternative. Dependent data - 2012 SIPP-EHC 6
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS SIPP Re-engineering Field Test Plans - EHC Proof of concept test - 2008 paper and pencil reinterview test - SIPP-EHC CAPI Prototype test - 2010 Integrated Blaise and C# instrument prototype - SIPP-EHC CAPI Revised - 2011 test improvements to the wave 1 instrument, training, and expand sample to all regional offices. - 2012 test wave 2 activities/instrument, Dependent Interviewing (DI), examine movers and attrition issues, and refine training procedures. 2013 research wave 3 – continue evaluating and refining content and DI. - Continue addressing instrument length and wave 3 issues, like returning household members. Dependent data - 2012 SIPP-EHC 7
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS Challenges Faced Developing new technical capacity - Experimentation - Limitations - Evolution Crisis planning - Limited lead time / preparation - Changing goals and required flexibility New procedures - Training and acceptance - Development and refinement of procedures Dependent data - 2012 SIPP-EHC 8
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS SIPP – Sample Design • National panel survey – Since 1984 with sample size between about 11,000 and 45,000 interviewed households • The duration of each panel varies from 2½ yrs to 4 yrs • The SIPP sample is a multistage-stratified sample of the U.S. civilian non-institutionalized population • The survey uses a 4-month recall period – 3 interviews / year • The sample is divided into 4 rotation groups for monthly interviewing • Interviews are conducted by personal visit and by decentralized telephone Dependent data - 2012 SIPP-EHC 9
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS SIPP – Following Rules • SIPP is a true longitudinal survey that tracks people over time, with few exceptions. • Original sample members are interviewed every 4 months over the duration of the panel. • Children (under age 15) are followed only if they move with an original sample adult (age 15 and over). • The SIPP rules call for following original sample members who move, provided they are not institutionalized, do not live in military barracks, or do not move abroad Dependent data - 2012 SIPP-EHC 10
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS 11
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS Panel Surveys – Dependent Data & Seams • What are dependent data? • Why use dependent interviewing (DI)? – Bias • Seam • Recall – Burden • Interview length • Cognitive challenge – Data quality • Longitudinal consistency • Corrections / improvements Dependent data - 2012 SIPP-EHC 12
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS Panel Surveys – Dependent Data & Seams • What are dependent data? – Substantive responses from previous interviews fed forward into the question design and flow of a subsequent interview for the same respondents. • What is dependent interviewing (DI)? – Making use of prior knowledge (dependent data) to change the context of a question or question flow to assist the interviewer and respondent in recalling high quality and consistent data from one interview to the next. See: Jäckle, Annette. (2009) Dependent Interviewing: A Framework and Application to Current Research. In P. Lynn (Ed), Methodology of Longitudinal Surveys (pp.93-112). Chichester, West Sussex, UK: John Wiley & Sons, Ltd. Dependent data - 2012 SIPP-EHC 13
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS Panel Surveys – Dependent Data & Seams • Why use dependent interviewing (DI)? – Bias • Seam bias is the pattern observed in longitudinal data created by a tendency for estimates to change across the „seam‟ where successive panel interviews meet. The estimates of change across „seams‟ usually far exceeds the expected pattern of transitions represented in „non - seam‟ time points within a single interview. • Recall bias is another source of measurement error which can greatly benefit from the use of DI. – Failure to report transitions or statuses – Mistimed reporting – Reporting errors occurring more often early in a retrospective interview reference period than for the time closer to the interview. Dependent data - 2012 SIPP-EHC 14
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS SIPP 1984 labor force transition rates (Martini, 1989) 25 20 15 Month 1 to 2 Month 2 to 3 % Month 3 to 4 Seam 10 5 0 EU EO UE EO OE OU Rates computed by dividing the transition by the previous month total in that status. E.g.: Total N of people moving from E to U (month 1 to month 2) / Total E (month 1) Dependent data - 2012 SIPP-EHC 15
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS Moore, J. (2007) Seam Bias in the 2004 SIPP Panel: Much Improved, but Much Bias Still Remains. Paper presented at the US Census Bureau / PSID Event History Calendar Research Conference. Suitland, MD. Dependent data - 2012 SIPP-EHC 16
1. CONTEXT 4. SIPP-EHC 2. SIPP HISTORY AND DESIGN 5. CONSTRAINTS 3. PANEL SURVEY DESIGN – DEPENDENT DATA & SEAMS 6. EVALUATION PLANS Moore, J. (2007) Seam Bias in the 2004 SIPP Panel: Much Improved, but Much Bias Still Remains. Paper presented at the US Census Bureau / PSID Event History Calendar Research Conference. Suitland, MD. Dependent data - 2012 SIPP-EHC 17
Recommend
More recommend
