Families Blossom Evaluation Workgroup Update Evaluation Team Bethany Lee, PhD, Evaluation Director October 21, 2016
Meet the Title IV-E Waiver Evaluation Team Pam Freeman, PhD Sara Betsinger, PhD Bethany Lee, PhD Elizabeth Greeno, PhD Lead Cost Evaluator Lead Evaluator for Evaluation Director Lead Evaluator for EBPs CANS-F/ Trauma Eric Slade, PhD John Cosgrove Cost Consultant Rochon Steward Senior Data David Chen Research Analyst Maria Jose Horen Senior Data Specialist Analyst Research Supervisor Rachel Margolis Research Asst
Evaluation Components Increasing Evidence-Based and Promising Practices • Process Evaluation • Outcomes Evaluation Creating a Trauma-Informed System • Process Evaluation • Outcomes Evaluation Cost Analysis • Cost shifts • Change in total expenses Monitoring Trends in State-Wide Outcomes
EBP Models • DSS Service Models – Baltimore City: Solution Based Casework (SBC) • Parenting Models – Allegany County: Incredible Years (IY) – Anne Arundel County: Parent-Child Interaction Therapy (PCIT) – Harford County: Nurturing Parenting Program (NPP) • Child Mental Health/Behavioral Health Models – Anne Arundel County: Functional Family Therapy (FFT) – Baltimore County: Cognitive Behavior Therapy+ (CBT+) SafeCare Comparative Effectiveness Trial: Howard and Prince George’s Counties
Start Dates County EBP Start Dates Allegany Incredible Years June 2016 Anne Arundel Functional Family Therapy September 2016 Anne Arundel Parent-Child Interaction September 2016 Therapy Baltimore City Solution-Based Casework September 2016 Baltimore County CBT+ June 2016 Harford County Nurturing Parenting April 2016 Program
Process Evaluation • National Implementation Research Network (NIRN) guidelines • What planning, training, and other practices happened in efforts to implement the evidence based practice? [what did we do?] • Tracking of individuals served and trained • Fidelity to the models
Process data elements for each EBP Type of Data Element Specific Elements to Collect: Data Program - Staff Training Describe the number and type of staff involved in Level implementation (including demographics, education, and workplace characteristics) and the training they received. Practice- Eligibility Describe client eligibility to receive the new intervention, Level process for referring subjects for services, alternates to the new intervention (where applicable). Client- Participation Describe the number of children/families served and the Level type and duration of services provided. Practice- Fidelity Describe instruments to assess fidelity to the intended Level service model. Practice- Implementation Report any barriers during implementation, steps taken to Level address the barriers, and lessons learned.
Outcome Data elements for each EBP Domain Data Element Specific Elements to Collect: Safety, MD CHESSIE Entry, re-entry, length of stay, permanency Outcomes placement type, stability; Well-being, Clinical Standardized measures specific to functioning outcomes the intervention Satisfaction Consumer Satisfaction with services, Feedback perceptions of impact Service use Auxiliary Public Public mental health claims or System Data (if juvenile justice involvement relevant)
Baltimore City: SBC • Sample: All 100+ Baltimore City In-Home Family Preservation workers will be trained in SBC. Estimated 2,000 families will be served with SBC per year. • Implementation Status: Train-the-training trainings occurred April-August (4 separate trainings). Training for in-home workers is slated for September 12 th to October 16 th . Trainings conducted by a certified SBC trainer.
Baltimore City: SBC • Design: One-group longitudinal design. • Measures: UMB SSW has collaborated with SBC providers to track implementation and fidelity. Online surveys have been given to Baltimore City staff to assess job-related quality of life (Professional Quality of Life), job burnout (Maslach Burnout Inventory), and Supervision Satisfaction (subscale of organizational climate measure). Baseline measures; posttest will be given between 12-18 months post training. • Secondary data from MD CHESSIE to monitor investigations, substantiations, or removals will occur semi-annually and during the 12-months following SBC participation.
Allegany County: IY • Sample: Any family with children ages birth to 12 that have involvement with Allegany Co DSS. Following IY guidelines for group size, first year approximately 30 families and 50 children will be served. • Implementation Status: IY currently implemented in collaboration with Family Junction. First group was completed end of September 2016.
Allegany County: IY • Design: 3-Group Quasi-Experimental Design to compare families and children receiving IY services to those receiving PCIT, and those receiving TAU from a comparable Maryland jurisdiction. Primary data will be collected at 2 time points: baseline and posttest. • Measures: Primary outcome measures will include child behavioral change (Eyberg Child Behavior Inventory), parenting stress (Parenting Stress Index). Additional measures will include therapeutic relationship alliance (Working Alliance Inventory-Client). • Secondary data from MD CHESSIE to monitor investigations, substantiations, or removals during the 12- months following IY participation.
Anne Arundel County: PCIT • Sample: Children ages 2-7 with severe behavioral health problems and their parents, who are receiving in-home services in Anne Arundel County. In year 1, approximately 25 children will be served. • Implementation Status: A PCIT provider has been identified and it is estimated PCIT will begin in September 2016.
Anne Arundel County: PCIT • Design : 3-Group Quasi-Experimental Design to compare families and children receiving PCIT services to those receiving IY, and those receiving TAU from a comparable Maryland jurisdiction. Primary data will be collected at 2 time points: baseline and posttest. • Measures: Primary outcome measures will include child behavioral change (Eyberg Child Behavior Inventory) and parenting stress and attitudes (Parenting Stress Index – Short Form). Additional measures will include therapeutic relationship alliance (Working Alliance Index – Client Version) and quality of family life (asked via open-ended questions). • Secondary data from MD CHESSIE to monitor investigations, substantiations, or removals during the 12-months following PCIT participation.
Harford County: NPP • Sample: Any family in Harford County who has an open child welfare case and has a child between the ages of 5 to 11. In year 1, 2 NPP groups will be completed with estimated 14 families and 20 children served the first year. • Implementation Status: NPP provider training of facilitators occurred in January/February 2016. First NPP group occurred April-June (8 participants total). Second cohort began in August 2016.
Harford County: NPP • Design: One-Group baseline-posttest design • Measures: Primary outcome data collected at 2 time points: baseline and posttest. As per the NPP model, baseline-posttest measures include the Adult Adolescent Parenting Inventory-2 (AAPI; measures parenting and child rearing attitudes) and the Nurturing Skills Quiz (measures quality of family life). Additional measures will include therapeutic relationship alliance (Working Alliance Inventory- Client). • Secondary data from MD CHESSIE to monitor investigations, substantiations, or removals during the 12-months following NPP participation.
Anne Arundel Co: FFT • Sample: Families with youth ages 11-18 years receiving in-home services, in a voluntary placement, or older adolescents returning home, presenting with behavioral issues. In year 1, approximately 15 families will be served. • Implementation: FFT provided by Center for Children, beginning around September 2016.
Anne Arundel Co: FFT • Design: Clinical outcomes of FFT measured by a Single Group Baseline-Posttest Design. – Quasi experimental design used to measure child welfare and juvenile justice outcomes. FFT youth compared to similar youth receiving TAU • Measures: Primary outcome measures will include clinical measures that will assess the youth’s moods and behaviors (Youth Outcome Questionnaire; Youth Outcome Questionnaire – Self- Report) and change in family functioning (Client Outcome Measure – Parent; Client Outcome Measure – Adolescent) from perspective of the parent and youth; therapist outcome measure. • Secondary data from MD CHESSIE to monitor investigations, substantiations, or removals during the 12-months following FFT participation.
Baltimore County: CBT+ • Sample: Two cohorts, each of 40 community MH providers and 40 CW staff will be trained; 80 families from each cohort expected to be served. – For CW staff: In-Home Service workers will be the primary group trained. For children/youth: any child over the age of 5 w/o cognitive limitations. • Implementation Status: Training is provided by the National Center for Evidence Based Practice in Child Welfare (NCEBPCW) at the UMB. – Training for the 1 st cohort occurred in June 2016; 36 mental health providers and 41 child welfare staff in attendance from THRIVE and Catholic Charities – Second training will occur in the fall
Recommend
More recommend
