Context Child welfare is turning to implementation science to - PDF document

The Role of Implementation Drivers in Child Welfare Systems Change David Lambert, PhD Tammy Richards, MEd Trish Knight, MPP Children’s Mental Health Research & Policy Conference March 3-5, 2014 Tampa, Florida Context • Child welfare is turning to implementation science to improve services and outcomes to children and families. • From 2008- 2013, the Children’s Bureau (CB) funded 5 national implementation centers (ICs) to develop & apply implementation knowledge within child welfare. • CB & ICs guided by National Implementation Research Network (NIRN) Framework, posits key stages and drivers important in successful implementation. 2 1

Context NIRN framework developed primarily by examining education and mental health programs  Tend to be service-level interventions and evidence- based Child welfare system change implementation projects tend to span service & system levels  Require more design & development than evidence- based service level interventions 3 Context Successful adaptation of NIRN to child welfare systems change requires understanding: • Which drivers are most important at different stages of implementation? • What is the child welfare context in which drivers are understood and used? 4 2

Research Questions • What do the phases of implementation look like in child welfare organizations that are designing change initiatives to improve practice? • Which drivers are most important to implementation projects at different stages of implementation? • What are the key components of these important drivers? • How much do important implementation drivers overlap with each other? • How do other state child welfare initiatives influence the implementation of systems change projects? Does this change the importance or context of key implementation drivers? 5 Implementation Drivers (“Keys”) An initial review of implementation literature identified “Keys to Implementation” in child welfare. These keys are very similar to, but not exactly the same as, NIRN drivers • Executive Leadership to establish vision, create urgency, and authorize managers to implement. • Stakeholder Involvement throughout the process. • Family Engagement in project design and implementation. • Culture and Climate should be assessed to understand potential readiness, support, acceptance, and resistance. • Communication to provide timely, clear, and consistent information about the project. • Cross-Functional Project Team is created to guide the project, manage details, and solve problems that may impede success. 6 3

NCIC Implementation Drivers (“Keys”) (continued) • Staff Competency: Coaching and Training helps staff assimilate training and use it in everyday practice. • Operational Details need to be thought through to align policy, training, and quality assurance. • Organizational Structures should align to promote horizontal and vertical integration. • Resources must be secured for additional and necessary equipment, materials, expertise and training. • Ongoing Commitment of leadership and resources is needed to support and sustain necessary project activities and change. • Evaluation and Monitoring is needed to assess and guide progress, understand fidelity, and assess impact. 7 Implementation Projects New Jersey: New York: New Hampshire: Manage by Building a Design and Data Project System of Development Sustainable Supports of a Practice for Child Welfare Model Supervisors  Build capacity to use  Provide system of  Improve consistency in family data to manage supports for child engagement and service welfare supervisors in a delivery  Developed and county administered,  Designed statewide practice implemented a model state supervised system to train Data Fellows model using inclusive design  Developed and using a competitive process involving Executive selection process implemented a model of leadership and field supervision piloted in 6  Training spanned 18  Model ultimately of the state’s 62 months, involved incorporated practice-based counties working with real data components including  Plans to expand the from their district on Solution Based Casework issues identified as a model statewide, based priority by managers. on what is learned. 8 4

Approach and Methods Interviews with agency staff involved in their state project on the implementation process • Interviews conducted three times: early, mid and end of project • Which drivers have been most important in implementing the project? • States describe drivers in their own words Detailed interview notes, including quotations, were coded using pre-established codes (implementation keys and NIRN phases of implementation framework) • Check-coding used to ensure inter-coder reliability and definitional clarity • Content analysis of interviews used to supplement NVivo coding query analysis 9 Caveats/Issues • Assessment of CW implementation is based on a small sample • Timing and length of time spent in the design/installation phase resulted in more data on design/installation, less on other phases. • While very important, “resources” key is likely over - represented due to interview instrument focus on the quality and usefulness of resources. • Ethnographic approach. Asks “which drivers were most important”, “what does this driver mean to you”, “how did it play out” and “what other factors in your state’s environment affected implementation?” 10 5

What do the phases of implementation look like in child welfare organizations that are designing change initiatives to improve practice? Phases of Implementation Exploration Design/Installation Implementation • Identify priority issue • Design of • Pilot selection Intervention • Consider potential • Begin roll out (build (curriculum, delivery solutions that fit with staff competency, method, develop agency context implement supporting model, policies that organizational • Identify general support practice) structures) strategy • Detailed Planning for • Implement Evaluation • Determine “design of Roll Out (timelines, and QA Structures to design” resource needs, provide feedback communication • Further design strategies, evaluation articulation as needed planning) 11 Keys and their Relative Importance over Phases of Implementation Exploration Installation/ Design Implementation Phase Phase phase Communication Cross Functional Project Team Culture and Climate Family Engagement Frequent Monitoring and Evaluation Leadership Commitment & Executive Sponsorship Ongoing Commitment Operational Details Organizational Structures Resources Staff Competency Coaching and Training Stakeholder Involvement Key never/ Key sometimes Key frequently Key consistently rarely described described described described 12 6

Which drivers are most important to implementation projects at different stages of implementation? Important Drivers over Phases of Implementation Exploration Design/ Installation Implementation • Resources • Resources • Resources • Culture and Climate • Stakeholder • Culture and Climate Involvement • Leadership & • Staff Competency: Executive Sponsorship • Culture and Climate Training and Coaching • Stakeholder • Organizational • Leadership & Involvement Structures Executive Sponsorship • Monitoring & • Operational Details Evaluation • Frequent Monitoring • Monitoring & and Evaluation • Operational Details Evaluation • Organizational Structures 13 Exploration A former CW Director • Time to reflect: current status, priorities, that could talk through benefit/challenges of taking on a project the process of • Consultation/knowledge transfer: impact of implementing a practice Resources culture and climate, logic model development, model was helpful as it was a peer that could change management, budgeting, staffing speak of their • Peer connections/Peer knowledge experience • Pre-existing agency culture is an important That’s what we see influencing the building block in initial conceptualization Culture and success of our • Does the current culture/climate support initiatives over the taking on a project? Climate years, that county organizational culture • How will culture climate support – or challenge and climate – a potential project? the Deputy Commissioner had worked on an earlier • Alignment of leadership’s goals/priorities with effort on supervision, Leadership the project focus so it’s an issue that is near and dear to her 14 7