Following 12 Large-Scale Implementation Initiatives Amy D. - PowerPoint PPT Presentation

A Mixed-Methods Study of System-Level Sustainability of an Evidence-Based Practice Following 12 Large-Scale Implementation Initiatives Amy D. Herschell, Ph.D., Ashley Tempel Scudder, Ph.D., Sarah Taber-Thomas, Ph.D., & Kristen M. Schaffner, Ph.D.

Current Team

PCIT Across PA Alumni

A Statewide Trial to Compare Three Training Models for Implementing an EBT Funder: NIMH R01 MH095750 Start Date: 9/18/12 Pennsylvania Pittsburgh- Stakeholders Project Length: 5 years based Team and Steering Committee Project PI: Amy Herschell, Ph.D. Project Coordinator: Shelley Hiegel, M.Ed. Science PCIT Experts Experts Project Trainers: Ashley T. Scudder, Ph.D. Sarah Taber-Thomas, Ph.D. Kristen F. Schaffner, Ph.D. NCSP

2011 Pennsylvania Agencies Providing Parent-Child Interaction Therapy

Present Pennsylvania Agencies Providing Parent-Child Interaction Therapy Last Updated – November, 2015

Statewide Steering Committee Membership Considerations  Child Serving Systems  Child Welfare  Education  Juvenile Justice  Mental Health  Regions of the State  Urban/Rural  Ultimate Goal - Representative from any group affected by PCIT Implementation – including Consumers

Steering Committee Members Parent of children Program Specialist Director of Pupil Services Parent who has with BH concerns Supervisor & Special Programs completed PCIT State Child Welfare School District Communications Consultant State OMHSAS Director Penn State EPIS Center President & CEO Behavior Health MCO Executive Director Community-based MH Agency Statewide Child Psychiatric Consultant OMHSAS Children’s Policy Specialist Pennsylvania Community Clinician Providers Association Community-based MH Service Systems Specialist Agency Behavioral Health Alliance ECMH Project of Rural Pennsylvania Manager Senior Medical Director Pennsylvania Key Behavioral Health MCO Professor of Practice Improvement Specialist, Psychiatry, Director, Evidence-based The Pennsylvania Child Welfare Psychology & Practice and Resource Center Pediatrics Innovation Center (EPIC)

Sustainability Defined  Sustainability begins 2+ years after implementation (Stirman et al., 2012)  Removal of formal training supports  Focus on fit into the environment  Generally thought to be the process of maintaining or improving a system’s ability to preserve a program’s function and utility

Sustainability Outcomes  Sustainability is tricky  75% have partial or low sustainability (Stirman et al., 2012)  REALLY TRICKY at the Client Level  Proportion of sites or providers sustaining (89%)  Proportion of eligible patients receiving an intervention after the period of training and implementation (11%)  EVEN TRICKIER at the State Level  Lower level of sustainment at the state level compared to the community level (Luke, 2014)

Dynamic Sustainability Framework

Current Study Aims 1. identify and evaluate specific methods used to sustain PCIT in service systems across the United States 2. Examine factors promoting or hindering long-term sustainability of PCIT following large-scale implementation initiatives

Inclusion Criteria  Large scale training initiative :  Training efforts across several counties or systems  Sustaining Phase of Implementation  Implementation Period: Prior to and during the formal 12 month training period (e.g., training, consultation)  Sustainability Period: Following the removal of initial formal implementation supports (e.g., training, formal consultation, or funding)

Identification of Initiatives  How?  Database Searches: PsycINFO, Academic Search Premier, Google Scholar  Search Terms: evidence-based practice, evidence-based treatment, dissemination, implementation, Parent-Child Interaction Therapy, sustainability, sustainment  PCIT Trainer and the treatment developer

Identification of Initiatives  How Many?  21 identified  6 did not meet “large scale” definition  2 were not yet in the sustaining phase  2 were really part of one initiative  12 initiatives included from 13 states

Large-scale PCIT Initiatives Enrolled 1. California 2. Delaware 3. Iowa 4. Michigan 5. Minnesota 6. Nebraska 7. North & South Carolina 8. Oklahoma 9. Oregon 10. Pennsylvania 11. Tennessee 12.Washington

Measures & Procedures  Implementation and training survey (25 items)  Training resources, Implementation Timeline, Subsequent Training Efforts, Methods & Consultation Approach  Sustainability interview  Open-ended  Active strategies, Barriers & Facilitators  Barriers, Strategies and Sustainment Survey (19 items)  Extent to which barriers were present and the degree to which specific strategies were used; overall level of sustainment  Program Sustainability Assessment Tool (40 items)  Environmental Support, funding stability, partnerships organizational capacity, program evaluation, program adaption communications, and strategic planning

Participants  12 initiatives  37 individuals participated in the sustainability interviews (M= 3.33, range = 1 to 8)  10 PCIT trainers  9 state officials  5 behavioral health providers  4 individuals from private foundations  2 directors of oversight centers  1 judge  1 managed care representative  5 individuals working in academic settings

Codebook Development Training and Intervention Strategies to Sustain System, Agency, and Funding Implementation Characteristics Therapist Factors Factors Openness/ Resistance to Approach/ Appeal of PCIT Federal Funds Infrastructure EBPs Philosophy Policy Trained Clinician Cost of PCIT State Funds Marketing Characteristics PCIT Champion Training and Cost-Benefit of Local Funds Integration into Implementation PCIT Existing Practices Factors Beyond the Agency Approach/ MCO Funds New Settings/ Support (+/-) Philosophy Populations Agency Support (+/-) Trained Clinician Private Insurance Funds Balancing Supply Characteristics and Demand Therapist Support (+/-) Training and Other Funds Continuing Implementation Education Factors PCIT Service Reimbursement Within Agency Training Building Partnerships Fidelity Monitoring Tracking Clinical Competency Monitoring Clinical Outcomes

Descriptives Min Max M SD % Clinicians Continuing to Provide 41 93 76.97 16.006 % Agencies Continuing to Provide 55 100 86.54 14.412 Total Clinicians Trained 27 >400 167.67 123.483 Self-Report of Overall Sustainability 2 7 5 1.537 PSAT Average 2.78 5.80 4.5229 .91853 Environmental Support 2.20 6.40 5.1667 1.18424 Funding Stability 2.00 5.60 4.2833 1.00348 Partnerships 2.40 7.00 4.6500 1.38334 Organizational Capacity 2.20 6.20 4.1333 1.30547 Program Evaluation 1.00 6.60 4.3167 1.75076 Program Adaptation 2.40 7.00 5.2333 1.15312 Communications 2 7 4.53 1.394 Strategic Planning 2.00 6.20 3.8667 1.22796

Model Summaries R 2 R 2 DV IV R changeR F chg p df 1 df 2 adj 2 Integration a % clinicians 1. .716 .512 .442 .512 7.344 .030 1 7 Integration a % agencies 1. .646 .417 .352 .417 6.444 .032 1 9 Integration a 1. Total clinicians .693 .480 .428 .480 9.224 .013 1 10 Integration b PSAT 1. .681 .464 .411 .464 8.663 .015 1 10 1. Barrier of financial .821 .674 .602 .210 5.805 .039 1 9 support b Integration b Overall 1. .904 .817 .798 .817 44.516 .000 1 10 Sustainability Monitoring quality b 1. .969 .939 .925 .122 17.917 .002 1 9 Note. * Indicates significance at p <.001. % clinicians =% clinicians continuing to provide, % agencies =% agencies continuing to provide, Total clinicians= Total clinicians trained, Overall Sustainability = Initiative Rating of Overall Sustainability. IV: a = Interviewer rating; b = Initiative Rating.

What seems to be the most important?  Integration (quant & qual)  Financing (quant & qual)  Quality Monitoring (quant & qual)  Intervention Champions (qual)

Integrating into practice  Training & education of other professionals (e.g., psychiatry residents, professionals in healthcare settings)  training in the basics of PCIT  educating other professionals about the intervention (e.g., Troutman, 2011).  Integrate PCIT into other services or programs,  Allowing PCIT to “bleed into a lot of other programs.”  Expand PCIT to new settings and populations  “special time” at a residential facility  Teacher-Child Interaction Therapy

Money Matters 2 subcategories  Implementation funding and financial support  Service reimbursement and billing practices “So, for us it is very costly to  “You can have the best do this stuff…. But then trained therapists in the these children’s trajectories  Incentives world but if nobody’s and their lives have  Increase funding dedicated to EBPs changed. So for me that’s going to pay for it then worth whatever you’re  Reimburse EBPs at higher session rate there’s not much putting into it.”  Recognize practices as EBPs incentive to keep  Service accommodations going…”  Disincentives  Existing/Competing services