Evaluating Complex Interventions Gareth Parry Senior Scientist, IHI Astou Coly June 27, 2017 Senior Improvement Advisor, URC
2 http://www.academyhealth.org/evaluationguide This report was made possible by generous support from the Robert Wood Johnson Foundation.
3 Background Focus on designs that are primarily intended to provide a quantitative estimate of the impact of an intervention Assumptions: – Evaluators and implementers have agreed on the overall theory of change – Key stakeholders have agreed on the evaluation questions Evaluability Assessment – Leviton LC et al Evaluability assessment to improve public health policies, programs, and practices. Annual Review of Public Health. 2010 Apr 21;31:213-33
Overview Experimental/Randomized Designs – Randomized Controlled Trial (RCT) – Cluster Randomized Stepped Wedge Design Quasi-experimental Designs – Interrupted Time Series Design (ITS) – Controlled Before and After Design – Regression Discontinuity Design – Before and After Study
5 Can randomize individuals to the intervention or control group? Yes Randomized Controlled Trial (RCT) Experimental/Randomized Designs
Experimental/Randomized Designs 6 Randomized Controlled Trial (RCT)
12-month outcomes of community engagement versus technical 7 assistance to implement depression collaborative care: a partnered, cluster, randomized, comparative effectiveness trial A cluster RCT conducted in Los Angeles, to compare the effectiveness of CEP vs TA approaches to depression collaborative care trainings Outcome: Mental health-related quality of life (MHRQL) and services use at 12-months. CEP was associated with a decrease in poor MHRQL compared to RS at 6 months (OR=0.71; 95% CI: 0.55-0.91) and 12 months (OR=0.77; 95% CI: 0.61-0.97). The authors concluded that while CEP did not indicate an effect at 12 months, policymakers and communities should still consider this strategy given the lack of alternative approaches that have demonstrated higher effectiveness. B Chung et al 12-month outcomes of community engagement versus technical assistance to implement depression collaborative care: a partnered, cluster, randomized, comparative effectiveness trial. Annals of internal medicine. 2014; 161(10 Suppl): S23-34
Experimental/Randomized Designs 8 Randomized Controlled Trial (RCT) Considerations “Gold Standard” Randomization allows for difference in outcomes between the intervention and control groups to be attributed to the intervention Randomization may not be practical or ethical Randomization at Cluster level Limited external validity Control group may be exposed to the intervention Retention of participants may differ between intervention and control groups Fixed-protocols limit adaptation of the interventions
Can randomize sites or communities to the 9 intervention or control group? Yes Cluster Randomized Controlled Trial (RCT) Experimental/Randomized Designs Must all sites or communities receive the intervention? Yes Cluster Randomized Stepped Wedge Design Experimental/Randomized Designs
Experimental/Randomized Designs 10 Cluster Randomized Stepped Wedge Design Time Period 1 2 3 4 5 6 7 8 9 10 Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 Site 7 Site 8 Site 9 Site not allocated to the intervention Site is allocated to the intervention
A structural multidisciplinary approach to depression management 11 in nursing-home residents: a multicentre, stepped-wedge cluster- randomised trial A stepped wedge design assessed the effectiveness of the Act in Case of Depression (AiD) approach to the management of depression among nursing home residents in the Netherlands All five groups received the intervention The authors concluded that while the AiD approach can reduce depression, screening needs to be addressed in dementia units as these units had lower adherence to screening Leontjevas, Ruslan et al. A structural multidisciplinary approach to depression management in nursing-home residents: a multicentre, stepped-wedge cluster-randomised trial. The Lancet, Volume 381 , Issue 9885 , 2255 - 2264
Experimental/Randomized Designs 12 Cluster Randomized Stepped Wedge Design Considerations Provides an alternative to the orthodox RCT approach All participants receive the intervention. Secular trends Evaluate barriers to implementation of the intervention and to iteratively improve implementation in subsequent steps. Preventing contamination between those receiving the intervention and those to receive the intervention may be particularly challenging. Requires frequent data collection.
13 Multiple data points available before and after the intervention Yes Interrupted Time Series Design Quasi-experimental Designs
Quasi-experimental Designs 14 Interrupted Time Series Design (ITS)
Association Between Hospital Penalty Status Under the Hospital 15 Readmission Reduction Program and Readmission Rates for Target and Nontarget Conditions As part of the Hospital Readmission Reduction Program (HRRP), financial penalties were imposed from Oct 2012, on hospitals with higher than expected readmissions. To assess the impact of the HRRP the investigators used an interrupted time series analysis design to compare trends in readmission rates between hospitals subject to and those not subject to the penalty. After the HRRP was announced, readmission rates declined more rapidly in hospitals later subject to penalties relative to those not penalized,. After implementation of the HRRP, the rate of change for readmission rates plateaued Before announcement of the HRRP The investigators concluded that the readmission rates were stable HRRP was associated with greater reductions in readmission rates in penalized hospitals relative to nonpenalized hospitals. Desai NR et al. Association Between Hospital Penalty Status Under the Hospital Readmission Reduction Program and Readmission Rates for Target and Nontarget Conditions. JAMA. 2016 Dec 27;316(24):2647-56.
Quasi-experimental Designs 16 Interrupted Time Series Design (ITS) Considerations Takes into account underlying trends in the data Can provide a strong quasi-experimental alternative to randomization Can assess effect size, speed and sustainability of the intervention over time There can be challenges in identifying comparator groups that provide comparable data It may be difficult to collect sufficient data points before and after the intervention to be able to detect a change in slope Adjusting for patient-level characteristics can be challenging
17 Comparator Groups available? Yes Controlled Before and After Design Quasi-experimental Designs
Quasi-experimental Designs 18 Controlled Before and After Design
The Impact of Green House Adoption on Medicare Spending and 19 Utilization To understand the impact of the Green House (GH) nursing home model on Medicare spending and utilization. Medicare claims and enrollment data and a resident-level assessment data set to estimate the impact of GH on Medicare acute hospital, other hospital, skilled nursing facility, and hospice spending and utilization. 15 GH intervention nursing homes and 223 matched nursing homes Propensity scoring to weight the data in a way that approximated intervention and comparison groups, with similar organizational characteristics. The investigators conclude that the nursing homes who adopted the GH model did not realize Medicare savings and suggested new approaches to align financial incentives may be required. Grabowski DC et al The impact of Green House adoption on Medicare spending and utilization. Health services research. 2016 Feb 1;51(S1):433-53..
Quasi-experimental Designs 20 Controlled Before and After Design Considerations Not subject to ethical and practical constraints of randomization Can be used in situations where it may not be possible to randomly assign the intervention The design is susceptible to bias and confounding which can be difficult or impossible to mitigate completely Many factors may influence exposure to “intervention” or “control” group. The reliance on existing data means that investigators may not have all needed data for the analyses
21 Is the intervention based on a cut off? Yes Regression Discontinuity Design Quasi-experimental Designs
Quasi-experimental Designs 22 Regression Discontinuity Design
Recommend
More recommend
