What works in Boston may not work in Los Angeles: Understanding site - PowerPoint PPT Presentation

What works in Boston may not work in Los Angeles: Understanding site di ff erences and generalizing e ff ects from one site to another. Kara Rudolph with Mark van der Laan RWJF Health and Society Scholar UC Berkeley / UC San Francisco Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 1 / 39

Outline 1 Motivation Motivating example 2 Methodologic Challenges 3 Approach 4 Results 5 Future directions Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 2 / 39

Motivation Should we expect that a policy/program/intervention implemented in one place will have the same e ff ect when implemented in another place? Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 3 / 39

Motivation Should we expect that a policy/program/intervention implemented in one place will have the same e ff ect when implemented in another place? Not always. Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 3 / 39

Motivation Should we expect that a policy/program/intervention implemented in one place will have the same e ff ect when implemented in another place? Not always. Di ff erences in site-level variables (e.g., implementation, economy) that 1 modify intervention e ff ectiveness, AND/OR Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 3 / 39

Motivation Should we expect that a policy/program/intervention implemented in one place will have the same e ff ect when implemented in another place? Not always. Di ff erences in site-level variables (e.g., implementation, economy) that 1 modify intervention e ff ectiveness, AND/OR Di ff erences in person-level variables (i.e, population composition) that 2 modify intervention e ff ectiveness. Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 3 / 39

Motivation Budgets are limited. Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 4 / 39

Motivation Budgets are limited. Need to target the policy/intervention to those places that stand to benefit most. Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 4 / 39

Motivation Budgets are limited. Need to target the policy/intervention to those places that stand to benefit most. E.g., planned expansion of intervention. Where should it be expanded to have the largest e ff ect? How is success defined? Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 4 / 39

Motivation Budgets are limited. Need to target the policy/intervention to those places that stand to benefit most. E.g., planned expansion of intervention. Where should it be expanded to have the largest e ff ect? How is success defined? Can you think of any practical examples of this? Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 4 / 39

Motivation Budgets are limited. Need to target the policy/intervention to those places that stand to benefit most. E.g., planned expansion of intervention. Where should it be expanded to have the largest e ff ect? How is success defined? Can you think of any practical examples of this? Research question: What do we expect the e ff ect of an intervention to be in a new place, accounting for population composition? Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 4 / 39

Outline 1 Motivation Motivating example 2 Methodologic Challenges 3 Approach 4 Results 5 Future directions Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 5 / 39

Motivating Example Moving To Opportunity (MTO) 1 http://www.chicagomag.com https://upload.wikimedia.org/ 1 Kling, J. R. et al. Experimental analysis of neighborhood e ff ects. Econometrica 75, 83–119 (2007). Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 6 / 39

Motivating Example In discussing di ff erences in e ff ects across sites, MTO researchers concluded: Of course, if it had been possible to attribute di ff erences in impacts across sites to di ff erences in site characteristics, that would have been very valuable information. Unfortunately, that was not possible. With only five sites, which di ff er in innumerable potentially relevant ways, it was simply not possible to disentangle the underlying factors that cause impacts to vary across sites. (This is true for both the quantitative analysis and for any qualitative analysis of the impacts that might be undertaken.) 2 Why are the researchers saying this? Do you agree? 2 Orr, L. et al. Moving to opportunity: Interim impacts evaluation. (2003), p.B11. Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 7 / 39

Motivating Example Research Question (MTO-specific): Are di ff erences in intervention e ff ects across cities due to di ff erences in implementation? City-level di ff erences (e.g, the economy)? Or di ff erences in population composition? Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 8 / 39

Outline 1 Motivation Motivating example 2 Methodologic Challenges 3 Approach 4 Results 5 Future directions Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 9 / 39

Methodologic Challenges Typically, multi-site data are analyzed using fixed e ff ects. Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 10 / 39

Methodologic Challenges Typically, multi-site data are analyzed using fixed e ff ects. Usually assumes that we answered “Yes” to whether we expect the intervention e ff ect in one site is the same as the other site. Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 10 / 39

Methodologic Challenges Typically, multi-site data are analyzed using fixed e ff ects. Usually assumes that we answered “Yes” to whether we expect the intervention e ff ect in one site is the same as the other site. Why is that the case? Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 10 / 39

Methodologic Challenges Typically, multi-site data are analyzed using fixed e ff ects. Usually assumes that we answered “Yes” to whether we expect the intervention e ff ect in one site is the same as the other site. Why is that the case? Dummy variables for site changes the intercept but not the treatment e ff ect coe ffi cient. Assume that the conditional e ff ect (regression coe ffi cient) of the intervention in one site is the same as in another site. Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 10 / 39

Methodologic Challenges Typically, multi-site data are analyzed using fixed e ff ects. Usually assumes that we answered “Yes” to whether we expect the intervention e ff ect in one site is the same as the other site. Why is that the case? Dummy variables for site changes the intercept but not the treatment e ff ect coe ffi cient. Assume that the conditional e ff ect (regression coe ffi cient) of the intervention in one site is the same as in another site. Need to apply the results from one city/site to a target city/site based on the observed population composition. Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 10 / 39

Methodologic Challenges Typically, multi-site data are analyzed using fixed e ff ects. Usually assumes that we answered “Yes” to whether we expect the intervention e ff ect in one site is the same as the other site. Why is that the case? Dummy variables for site changes the intercept but not the treatment e ff ect coe ffi cient. Assume that the conditional e ff ect (regression coe ffi cient) of the intervention in one site is the same as in another site. Need to apply the results from one city/site to a target city/site based on the observed population composition. Transportability/ generalizability/ external validity. Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 10 / 39

What’s been done Most common: Use fixed e ff ects for site. 3 Miettinen, O. S. Standardization of risk ratios. American Journal of Epidemiology 96, 383–388 (1972). Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 11 / 39

What’s been done Most common: Use fixed e ff ects for site. - Conditional e ff ect is not as policy relevant as marginal e ff ect 3 Miettinen, O. S. Standardization of risk ratios. American Journal of Epidemiology 96, 383–388 (1972). Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 11 / 39

What’s been done Most common: Use fixed e ff ects for site. - Conditional e ff ect is not as policy relevant as marginal e ff ect - We usually don’t believe the assumption 3 Miettinen, O. S. Standardization of risk ratios. American Journal of Epidemiology 96, 383–388 (1972). Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 11 / 39

What’s been done Most common: Use fixed e ff ects for site. - Conditional e ff ect is not as policy relevant as marginal e ff ect - We usually don’t believe the assumption Common-ish: Post-stratification/ direct standardization. 3 E.g., age-adjusted rates of disease for comparisons between populations. 3 Miettinen, O. S. Standardization of risk ratios. American Journal of Epidemiology 96, 383–388 (1972). Kara Rudolph (UCB/UCSF) Generalizing e ff ects across sites 11 / 39