Research: The Game Changers Mark Schulman, Ph.D. AAPOR Roundtable - PowerPoint PPT Presentation

Evolving Survey Research: The Game Changers Mark Schulman, Ph.D. AAPOR Roundtable 67th Annual Conference May 2012

From the great sage himself… “The future ain't what it used to be.” Yogi Berra US baseball player, coach, & manager (1925 - ) Abt Associates | pg 2

Challenges and Opportunities  Difficulties of securing cooperation in our busy and diverse society  Growing recognition of shortcomings in traditional self-reports research  Emergence of new technologies which are redefining what we can do and collect  Vast stores of real-time administrative records may be utilized by govt. in place of or augmentiing contracted surveys  Need for more efficient sample designs, particularly for rare populations  Probable decline of federal/state research budgets, boosting search for efficiencies Abt Associates | pg 3

Digital Revolution Opens New Doors  Smartphones, iPads, other digital devices and social media provide real- time interactions to capture consumer behavior, preferences, underlying motivational structures, diffusion, and even photographs.  Require new perspectives in the conduct of surveys, well beyond the stale "online vs. phone vs. household versus intercept."  Require multiple data collection modes to reach our diverse population and better understanding of mode effects  Need to combine multiple software platforms  Real data capture, not just self-reports on questionnaires! Abt Associates | pg 4

Smartphones will soon overtake ordinary cell phones Abt Associates | pg 5

Even Smarter Smartphones!  Talk to Siri as you would to a person. Say something like “Tell my wife I’m running late.” “Remind me to call the vet.” “Any good burger joints around here?” Siri does what you say, finds the information you need, then answers you. It’s like you’re having a conversation with your iPhone. Abt Associates | pg 6

How do we harness apps?*  Smartphones equipped with peripherals give researchers new ways to automate data collection and potentially reduce:  recall bias  noncompliance/item nonresponse  overall measurement error *(Buskirk, “Is your Health Survey Research as Smart as your Phone,” 2011) Abt Associates | pg 7

Changing World Example: Prevention and Treatment Apps in Public Health – How do we harness?* • Harvard’s iPhone Swine Flu Tracking app (Rao, 2009) • Apps to improve patient/provider communications (Patrick, et al. 2008) • Wellness diary apps for health promotion (Koskinen and Salminen, 2007) • Diabetes maintenance and smoking cessation apps (Logan et al., 2007 and Abroms, et al., 2011) *See, Trent Buskirk, “Is your Health Survey Research as Smart as your Phone?,” 2011 Abt Associates | pg 8

Example: Harvard Swine Flu App Includes videos, animations and text that allow you to learn: • how to reduce the risk to you and your family • how to prepare your business for the pandemic • real-time updates and news from Harvard Med about H1N1. •“HealthMap” about the state of the epidemic in your current location and other locations. Take away points for surveys? Abt Associates | pg 9

Smartphone Prevention and Treatment Apps  Wahoo’s Fitness Packs  ThinkLab’s Electronic Stethoscope (and Heart rate Monitoring) Abt Associates | pg 10

Digital measurement: new frontiers  Abt SRBI Transportation/Urban Planning Group pioneering in application of GIS to Household Travel Surveys  GIS devices supplementing paper-and-pencil diaries to measure all household travel  Old self-report diaries omitted much household travel Abt Associates | pg 11

Other Innovative GIS Applications  Agency for Healthcare Research and Quality (AHRQ). Abt SRBI providing GIS services to Abt Associates to develop a Learning Network and Technical Assistance Center to support principal investigators and their research.  Neighborhood Stabilization Tracking Panel (NSP). Abt SRBI GIS is working with Abt to manage all GIS mapping and analysis for the visual tracking survey component. Measuring potential impacts of a program designed to stabilize neighborhoods with high rates of foreclosures or vacancies.  NYC Survey of Youths. Abt SRBI GIS is developing live spatial analysis services for real time screening in CATI systems of survey participants. These live GIS services allow for customized geographies to be utilized screening by home locations in designated complex survey target areas.  Environmental Analysis - Digital Elevation Models (DEM), topography imagery analysis, view shed and watershed modeling, etc. Example: Advisory Base Flood Elevations (ABFE). Abt SRBI GIS supported Carlos Martín and Tom McCall of Abt Associates in Bethesda, MD in determining the Advisory Base Flood Elevations (ABFE) for 3,626 properties throughout Texas, Louisiana and Mississippi. Abt Associates | pg 12

Sampling/Stat: New Frontiers  Increasing use of efficient sample designs, such as dual ABS - List  Current early-stage explorations of modeling techniques for non-probability samples: probability of selection not known -- such as web panels  Early attempts: Propensity Scoring: Other adjustment and modeling techniques being explored to make inferences from non-prob samples.  Growing use of imputation to substitute for missing data.  Growing use of paradata and responsive design: Data about the process by which the survey data were collected. Paradata includes the times of day interviews were conducted, how long the interviews took, how many times there were contacts with each interviewee or attempts to contact the interviewee, the reluctance of the interviewee, and the mode of communication (such as phone, Web, email, or in person). Responsive design to optimize sample.  Respondent-driven sample: variant of snowball sampling. Not in favor. Abt Associates | pg 13

Data Collection from Administrative Records  Government records contain huge amounts of data which may some day supplant the need for contracted surveys.  Govt. data bases include Medicare, Medicaid, Social Security, birth/death records, etc.  Netherlands census does not conduct a total enumeration, but relies on administrative records and limited sample surveys  U.S. Census examining this approach. See Bob Groves’s blogs  Very major privacy issues and “big brother” political issues here.  May be appropriate mianly for govt.-conducted surveys.  SRBI has a medical records coding staff in place. Abt Associates | pg 14

Biospecimen Collection  Growing use of biological specimen collection in surveys.  Combine biological specimens with social science data  Includes blood, saliva, etc.  SRBI collecting specimens for surveys of military veterans, urine specimens, etc. Abt Associates | pg 15

Biospecimen Collection*  Combining biospecimens with social surveys adds complex legal/ethical/privacy issues and cost to projects.  Storing specimens adds additional issues, including storage, retrieval, and sharing of data.  Informed consent becomes much more complex. (See Committee on National Statistics Report, 2010) Abt Associates | pg 16

Market Research: Changing Landscape  Social media, smart phones and other digital devices provide real-time interactions to capture consumer behavior, preferences, their underlying motivational structures, and diffusion.  Tweets becoming social sensors, measuring everything from purchase behavior and movie preferences to influenza rates  Applications when don’t need accurate estimation, such as qualitative  My new years resolution: Never be afraid to be kicked in the teeth. Let the blood and the bruises define your legacy. - Lady Gaga 2012 1 Jan Abt Associates | pg 17

Google Flu Trends 2007-2008  Close relationship between how many people search for flu-related topics and how many people actually have flu symptoms (CDC). Blue: CDC Yellow: Google Estimate Abt Associates | pg 18

Web Spying!  Web surveillance, with strategic partners, tracks and analyzes millions of online tweets, postings, chat rooms, etc. http://www.radian6.com/see- demo/?utm_source=google&utm_medium=adwords&utm_campaign=socialmediamonitoring&gclid=CPCW 54eZt60CFUOQ7QodI2TFoA Abt Associates | pg 19

Bottom Line  Need to leverage growing technological innovation for data collection.  More development and application of efficient sampling More expansive definition of “data collection,” including  biospecimens  New technologies and apps require broader thinking  Explore greater use of administrative records in place of or in conjunction with surveys for some government projects Abt Associates | pg 20

Discussion Abt Associates | pg 21