1
First, a quick overview of the presentation: • I’ll describe the multimode survey that formed the base of this analysis. • I’ll talk about the yield that came from different types of extra effort. • I’ll show the effect that extra telephone effort had on both mail and phone completes. • I’ll show how this varied by English versus Spanish language interviews, • And examine how extra effort affected survey demographics. 2
This was a survey of SNAP participants – this is the program run by the US Department of Agriculture’s Food and Nutrition Service, it used to be referred to as the “Food Stamps” program. The sample for the study was provided by 26 different state SNAP programs. Those who completed were promised a $20 cash incentive. The purpose was to understand the food and shopping habits of SNAP recipients as well as their knowledge of healthy eating – to address questions such as… Where do SNAP recipients shop? Why do they choose these locations? What types of foods do they buy? Why? Do they think their diets are healthy? Why or why not? What barriers prevent them from having a healthy diet? 3
The survey was primarily a mail survey, with multiple mail contacts. Spanish language surveys were only provided upon request. The phone center conducted telephone followup for mail nonrespondents, which started 7 weeks after the first mailing. Participants were encouraged to complete the survey by phone, or to complete the mail survey. The phone survey response rate was 10%, this reflects just phone completes from calling, and remember this was the sample that had failed to complete the mail survey after 4 different mail contacts. The overall survey response rate was 40%. The analysis for this presentation arose out of an unexpectedly large time gap between the first sample release and the reserve sample. This initially was set to be 4 weeks, but changed to a 9 week gap. This presented an opportunity to utilize all possible “extra efforts” available to us, and this was necessary in order to retain staff/keep the callers busy while waiting for the new sample to arrive… 4
So what are these “extra effort” techniques? Refusal conversion is typically employed for most studies – we typically wait 2 weeks after the initial refusal to attempt refusal conversion. In our calling system, repeat non-contacts (ring no answer, or answering machine/voice mail results) finalize after 7 calls. We can elect to put them back out for additional calls, and also typically hold these for about a week before doing so. Similarly, cases with which we have made contact, but have not completed or otherwise finalized, are coded as maximum calls after 9 attempts – these can also be refielded for additional calls after a rest period. And finally, tracing is something we employed on this study. The purpose was to identify better phone numbers for cases that we called but reached either nonworking numbers or working numbers at which the sampled person could not be reached. Given the nature of this study’s population (being very low income, and highly mobile in terms of living arrangements), this was an important effort to employ on this project. Some cases ended up with combinations of the above approaches, and this group is shown separately in my analysis from those who only had one type of extra effort. The focus of this presentation is on completed interviews (by mail or phone) among the body of cases sent to the phone center for phone followup, and on the contributions of these extra efforts to survey response. 5
It’s interesting to look at how much of the sample was eligible for some type of extra effort. Out of our phone sample of 6,776 cases, about 3/4ths of the cases were eligible. The most common type was non-contact refielding at 19%, followed by tracing at 16%, then refusal conversion, and then maximum calls refielding. Cases eligible for combinations of the above represented 1 out of 5 cases in this sample. 6
Overall, what did extra effort yield in terms of survey completes? • The first thing of interest on this slide is to see how many completes came in by mail, among the cases we called. Nearly as many as by phone (595 vs. 611)! • Also, the dynamics of which extra efforts were effective in terms of yielding completes varied substantially by phone versus mail. • More phone completes than mail completes required extra effort • Refusal conversion and maximum call refielding yielded more phone completes than mail. • Noncontact refielding and tracing yielded more mail completes than phone. • Without these extra efforts, one-fourth or more of completes would not have been obtained. 7
How did this play out by language of interview? First, I’ll show the English language completes. This slide looks very similar to the previous slide, as these cases represent the majority of the sample. What’s interesting here is that our phone prompting yielded more mail completes than phone completes (568 vs. 484). 8
When we add in the analysis for the Spanish language interviews, the picture changes dramatically. • Here, phone prompting yielded far more phone completes than mail completes (127 versus 27). • Also, the pattern of which extra efforts were most effective changes. • For Spanish-speakers, refusal conversion was less effective than for English speakers. • Non-contact refielding and tracing were also not very effective for this group. • Maximum call refielding, however, was very effective in terms of yielding phone completes and mail completes. 9
Now we look at the demographics of cases that completed the interview by phone, comparing those that were obtained via extra effort to those that required no extra effort. For this analysis, I’ve restricted to looking just at phone completes – since I only had demographics for those cases in my analysis file. No significant differences were observed when looking at extra effort versus none by: • Gender • Education • Housing tenure • Self-reported health status • And household composition, including: • One adult households versus all other types • Household with children versus all other types 10
The only demographic comparison that yielded even marginal significance was looking at Hispanic/Latino versus not Hispanic/Latino. These results are very similar to the comparison of Spanish versus English language interviews. For all Hispanics (not just Spanish speaking), refusal conversion and maximum call refielding were equally effective. 11
And, when we looked at those born outside the US versus those born in the US, we see a similar picture as for Spanish speakers and Hispanics. Refusal conversion and Maximum call refielding are the most effective “extra efforts” for this group as well. 12
Although the analysis by age group was non-significant, it does raise some interesting findings. Of note here is that the two groups that were least likely to require extra effort were the oldest and the youngest. Just 20% of completes by 18-29 year olds, and 25% of those 60 and older, were obtained via extra effort. For all other age groups, the rate was 30% or higher. 13
Analysis of phone completes by whether the household reported currently receiving SNAP versus not receiving SNAP yielded the only significant result among the demographic comparisons. Here, nearly half of completes from those not currently receiving SNAP were obtained via extra effort, compared to just 1/4 th of completes for those receiving SNAP. These completes were more likely to require refusal conversion and noncontact refielding than those currently receiving SNAP. This finding suggests that salience was a key factor in response to this survey. 14
In conclusion: Extra efforts were useful in both generating additional phone completes as well as encouraging mail response for this survey. Spanish-speakers responded to different stimuli (among the types of extra effort) than English speakers. Similar patterns were observed for Hispanics and for those born outside the U.S. Revisiting maximum calls cases is very useful for these populations The salience of the survey request (whether currently receiving SNAP) had the strongest effect in terms of whether extra effort was required, and the type of extra effort. Non-significant findings were observed for other demographic comparisons. 15
A few words about the implications of this research as far as cost efficiency: • Among the forms of extra effort described here, tracing is the most costly approach. This requires 2 kinds of labor, both the basic operation as well as the efforts by the tracing team. • On this study, tracing was discontinued as of the time the reserve sample was deployed, in order to preserve hours for the main calling effort. • The other types of efforts are less expensive, in that they only require basic interviewers and extra calls and time spent on outstanding cases. In terms of limitations: • This analysis represents just one study. As our phone center handles mainly specific list sample designs these days, I intend to extend the analysis to other populations in the future. • The sample represents a very specific population, one that is very low income and highly mobile. Results will likely be different when looking at other types of samples. • And finally, this was an organic analysis, not a planned experiment with an experimental design. 16
I will be happy to address questions about this research. My email is shown here. 17
Recommend
More recommend