Connec Con necting: ting: Adapting Survey Methods to Reach a - PowerPoint PPT Presentation

Connec Con necting: ting: Adapting Survey Methods to Reach a Highly Mobile Sample Pr Presentation ion at the 20 2014 AA AAPO POR Co Conference Anaheim, , Ca Cali lifornia ia May 16, 2014 Melissa Dugger

Presentation Overview • Study background • Sample • Data collection • Locating • Outcomes • Lessons learned 2

A Vision, Strategy, and Evaluation • San Francisco-based intermediary with strategic vision • Strategy: social enterprise employment • Evaluation culture • Key interests: Employment, housing, criminal activity 3

Social Enterprises (1) Annual Social Business Employment Year Organization Enterprises Line (approximate) Started Target Population Organization A Blue Skies Cafes 18 1986 Mental health disabilities Cleaning 23 2009 People’s Food 0 2013 Harvest processing Organization B Golden Street 108 2011 Parolees State Works cleaning Organization C Chrysalis Staffing 500 1984 Formerly Enterprises incarcerated, Street homeless cleaning 4

Social Enterprises (2) Annual Social Business Employment Year Organization Enterprises Line (approximate) Started Target Population Organization D SF Solutions Lobby 55 2007 Homeless services Maintenance 30 services Organization E Back-to- Retail 36 2012 Any barrier Work Organization F Hope Construction 12 2007 Young adults aged 18 – 28 who are not Builders enrolled in school or participating the labor market Organization 360 ° Pest control 10 2012 Homeless G Solutions 5

Data Collection Study Completion Component Interviewer Administration Start Date Date Incentive Baseline Social When 4/1/2012 3/31/2013 Study survey enterprise participant brochure and staff member enters the social magnet with Mathematica’s enterprise information Exit survey Social Six months after 5/22/2013 8/29/2013 $20 gift card enterprise participant staff member leaves the social enterprise Follow-up Mathematica Approximately 9/3/2013 3/14/2014 $20 gift card survey interviewer one year after participant leaves the social enterprise 6

Follow-Up Survey • Conducted approximately one year after the baseline survey • Hard copy survey was administered by phone by Mathematica phone interviewers Group Start Phone Calls 1 September 4, 2013 2 October 2, 2013 3 November 2, 2013 • Field locating started on November 15, 2013 • Data collection ended on March 14, 2014 • Hired four field locators 7

MJS Target Population and Samples Consent Work SE Sampled (85%) (87%) (95%) Follow-up Baseline Exit Target population survey survey survey population (n = 718) (n = 609) (n = 527) (n = 582) Not work SE (n=82) Source: Mathematica Jobs Study database 8

A Difficult Population to Follow All Org A Org B Org C Org D Org E Org F Org G Average 41 37 33 44 41 46 25 49 age Never 25 46 16 25 29 26 23 0 employed No high 29 10 51 27 17 6 38 0 school diploma Ever 69 25 100 71 61 20 62 20 convicted Homeless 85 84 90 83 90 77 38 100 Work 23 18 32 17 30 35 79 69 income Note: Homeless means did not stay in owned or rented housing for entirety of year before baseline survey; work income is the percentage of income from work. All numbers percentages unless otherwise noted. 9

Targets by Organization Expected Number of Cases Completes Organization A 41 23 Organization B 94 52 Organization C 356 196 Treatment 274 151 Control 82 45 Organization D 42 23 Organization E 31 17 Organization F 13 7 Organization G 5 3 Total 582 321 10

Two Tiers of Locating Level 1: Traditional locating Level 2: Integration of organizations 11

Level 1: Traditional Locating (1) • Obtained extensive contact information on baseline and exit surveys • Identified and updated contact information with “holiday cards” • Used locating resources, including Web searches and a search through a database of public records 12

Level 1: Traditional Locating (2) • Identified incarcerated individuals weekly and monitored release dates to remove individuals temporarily or permanently from survey efforts • Sent email to nonrespondents and refusal conversion letter to respondents who had refused • Deployed field locators • Heavily monitored staff productivity and locating efforts 13

Level 1: Traditional Locating (3) Completion Completion Locating Start Rate When Rate at the Date Started End Level 1 11/15/14 24% 39% (141) (231) 14

Level 1: Traditional Locating (4) Expected Total Completes to # of Completes Completes Date Needed Organization A 23 19 4 52 20 32 Organization B 196 152 44 Organization C 23 14 9 Organization D Organization E 17 17 0 Organization F 7 7 0 3 2 1 Organization G Total 321 231 90 15

Level 2: Integrated Organizations in Locating (1) • Hired organization staff as field locators • Had organizations provide updated contact information for nonrespondents, including current work status in the social enterprise • Provided staff at organizations with incentives • Integrated information about the follow-up survey into organization's support services (for example, retention staff reminded participants verbally or with written materials about the study and provided phone numbers, phones, and on-site gift cards for completing the survey) 16

Level 2: Integrated Organizations in Locating (2) Completion Completion Locating Start Rate when Rate at the Date Started End Level 1 11/15/14 24% 39% (141) (231) Level 2 01/28/14 39% 44% (231) (258) 17

Level 2: Integrated Organizations in Locating (3) Expected Total Completes to # of Completes Completes Date Needed 23 19 4 Organization A Organization B 52 20 32 Organization C 196 152 44 Organization D 23 20 3 17 17 0 Organization E 7 8 -1 Organization F Organization G 3 4 -1 321 258 63 Total 18

Assessment of Tier 1 & 2 Locating Effort • 11 weeks in the field • Current response rate 44 percent • 3 weeks left in the field period • 63 completes needed to reach target response rate • Average of 9 completes per week 19

Assessment of Locating Effort • Organizations use Facebook for outreach • One social enterprise participant blogged about their experience 20

Can We Use Facebook to Reach Out to Participants to Complete the Follow-Up Survey? 21

Relevant Resources • According to the Public Policy Institute of California, internet use among Californians ages 35 to 54 is 88 percent • In southern California, where there is a high number of nonrespondents, internet use was over 85 percent 22

Level 3: Using Social Media for Locating • Ask organizations to post a reminder message about the follow-up survey on their Facebook page • Use Facebook to find additional information that might facilitate locating • Send personalized messages to nonrespondents 23

Facebook Profile 24

Facebook Protocol: Match Information • First and last name • Date of birth • City and state • Email address • Family member • Mention of social enterprise 25

Facebook Outcome Number of Unable to Organizations Cases Name Email Locate Organization B 20 8 4 8 Organization C 127 36 23 68 TOTAL 147 71 76 *We only considered cases with an interim status code for Facebook locating. 26

Email or Name Match 80 68 70 60 Number of Cases 50 Email 40 36 Name 30 Unable to locate 23 20 8 8 10 4 0 Org B Org C 27

Email or Name and Other Information 70 6 60 50 Number of Cases 40 Other 30 59 Email or Name 20 3 10 12 0 Org B Org C 28

Locating Start Dates and Completion Rates Locating Start Completion Completion Date Rate When Rate at the Started End Level 1 11/15/14 24% 39% (141) (231) Level 2 01/28/14 39% 44% (231) (258) Level 3 02/20/14 44% 47% (258) (274) Completion rate is: # completed surveys/# follow up sample 29

Completes by Organization Expected Number of Completes Completes Organization A 23 21 Organization B 52 32 Organization C 196 170 Treatment 151 138 Control 45 32 Organization D 23 21 Organization E 17 18 Organization F 7 8 Organization G 3 4 Total 321 274 30

Response Rates Number Completed Response Surveyed Surveys Incarcerated Deceased Rates (%) Organization A 41 21 1 0 53 Organization B 94 32 20 1 44 Organization C 356 170 21 1 51 Treatment 274 138 16 1 54 Comparison 82 32 5 0 42 Organization D 42 21 2 0 53 Organization E 31 18 0 0 58 Organization F 13 8 1 1 73 Organization G 5 4 0 0 80 Total 582 274 45 3 51 Response rate = # completed/(# surveyed - # incarcerated - # deceased) 31

Lessons Learned • If you think you may want to implement social media into the locating effort, gain consent and important contact information at the beginning of data collection • Leverage the social media outreach from the organizations at the onset of data collection • Don’t discount using social media for hard -to-reach populations 32

For More Information • Melissa Dugger MDugger@mathematica-mpr.com 33