How to Continuously Evaluate System of Care Expansion by Combining Universal Big Data From All Human Services and Entire School Populations: Examples From Chautauqua & Rockland Counties (New York State) & Manchester City Council (UK) Workshop Presented at University of Maryland Training Institutes, July 25-28,2018, Washington, D. C. Mansoor A. F. Kazi, PhD, President, Realist Evaluation Inc., Director, Program Evaluation Center, Fredonia SUNY; Rachel M Ludwig, LCSW, Chautauqua Tapestry Director; John Rosiak, Rosiak Associates, LLC Susan Hoerter, DO, Rockland County SOC Expansion Grant PI; Janet Sliva, Rockland County SOC Expansion Grant Manager
Acknowledgements • Dr. Bret Apthorpe, Superintendent; Jessie Joy, Director of Curriculum, Instruction & Assessment; Tina Sandstrom, Director of Schools, Jamestown Public Schools; John Panebianco, Director of Pupil Services • Dr. Gary M. Blau, Chief, Child, Adolescent and Family Branch, Center for Mental Health Services, Substance Abuse and Mental Health Services Administration • Patricia Brinkman, Commissioner; Chautauqua County Department of Mental Hygiene, Mayville • Rachel Ludwig, Director Chautauqua Tapestry • Manchester City Council — Marie McLaughlin, Director Youth Justice Services
Acknowledgements • Ed Day, Rockland County Executive • Tom Zugibe, District Attorney • Michael Leitzes, Mental Health Commissioner • Mary Jean Marsico, Rockland BOCES • Dr. Susan Hoerter, Rockland SOC Expansion PI • Janet Sliva, Rockland SOC Expansion Grant Manager • Barbara Gavin, Director, Child Welfare Services • Carol Korngold, Safe Youth Center Executive Director • Kathy Farber, Safe Youth Center Consulting Director • Kathy Tower-Bernstein, Director of Probation
Mansoor A. F. Kazi, President, Realist Evaluation Inc.; Realist Evaluation Partnerships Affiliate Research Fellow, University at Buffalo; Director, Program Evaluation Center, Fredonia State University of Tapestry ry of New York; Adjunct Fordham University Chauta tauqu qua a kazim@fredonia.edu mkazi@buffalo.edu • County, ty, American Evaluation Association (eval.org) Co-Chair Human Services Evaluation Topical Interest Group & New York Social Work Topical Interest Group. SAMHSA’s Gold Award for MANCHESTER CITY Outstanding COUNCIL, England Local Evaluation 2010 Training ROCKLAND COUNTY NY Institute STAKES/THL Based on Kazi, M. A. F. (2003) ‘Realist Evaluation in Practice’, London: Sage
New York State System of Care SAMHSA Funding Funded by SAMHSA — Substance Abuse and Mental Health Services Administration, Center for Mental Health Services: • ‘Tapestry of Chautauqua’ Systems of Care Grant (with Chautauqua County Department of Mental Hygiene, $9,000,000 2008-2015; $977,317 for evaluation, Funded by SAMHSA; Recipient of SAMHSA’s Gold Award for Outstanding Local Evaluation, July 2010). • ‘Tapestry of Chautauqua’ Systems of Care Expansion Grant (with Chautauqua County Department of Mental Hygiene, $4,000,000 2015-2019) • Rockland County (NY) Systems of Care Expansion Grant, with Rockland County, $4,000,000 2016-2020; $800,000 for evaluation, Funded by Substance Abuse and Mental Health Services Administration (SAMHSA).
Take the Gary blau challenge
System of Care: Focus on Values Continuous Quality Improvement Cultural & Linguistic Family Driven Competence T ransforming Children’s EBP’s & T= (V + B + A) x (CQI) 2 Mental Health in America! Clinical Youth Guided Excellence
Children’s Mental Health Initiative (CMHI): Systems of Care Since 2000, SAMHSA administers an average of 61 system of care CMHI has grants per year impacted 22% of CMHI sites have the nations served over counties and 33 100,000 children federally and youth recognized Indian Tribes There are currently 88 CMHI funded sites
Systems Of Care Work! Reduced behavioral and emotional problems Increased behavioral and emotional skills Reduced suicidal ideation and attempts Reduced substance use problems Improved functioning in school and in the community Improved ability to build relationships
View all of the KSOC-TV webisodes on the SAMHSA YouTube channel or by going to www.samhsa.gov/children
The Goals of a System of Care • To improve access and expand the array of services and supports for children and youth with a serious emotional disturbance and their families. • To promote the full potential of every child and youth by addressing their physical, emotional, intellectual, cultural and social needs in family, school & community
Who Comes to School? • 21% of children ages 9-17 have a diagnosable mental health disorder (U.S. Surgeon General, 1999; NIMH, 20% currently) • 20% of children with mental health disorders are identified and receive mental health services (U.S. Surgeon General, 2000; mentalhealth.gov — 20% currently) • 50% of students age 14+ who have a mental illness drop out of high school…the highest dropout rate of any disability group (U.S. Department of Education, 2001) (NIMH currently)
Chautauqua Tapestry • Why “Tapestry”? • Quick history • Goals • Partners
SAMHSA’s Theory of Change
Expansion = System Change System of care expansion is not a “project” Goal is sustainable systemic changes in child-serving systems to improve services and outcomes Occurs with or without federal grant Grants are “venture capital” to achieve larger system change Linked to other system reforms
Keys to Successful Expansion 1. Commitment of high- 6. Linkages with larger level policy and decision system reforms in makers environment 2. Realistic goals 7. Cross-agency partnerships 3. High-priority goals 8. Commitment across key 4. Specific, concrete stakeholders strategies 9. Staff and resources 5. Strong leadership to allocated to manage implementation implementation work
Evaluation as Sustainability Strategy • Similar organizations collecting quality data, but uncoordinated and isolated • Similar organizations collecting quality data for the same outcomes in a coordinated and collaborative network • Similar organizations collecting quality data that align under similar outcomes for broad community impact
Collaboration Defined : A mutually beneficial relationship between two or more parties who work toward common goals by sharing responsibility, authority and accountability for achieving results.
New York State
Evaluation Examples • Jamestown Public Schools • North Rockland School District • Manchester City Council, UK
Critical Realism and Outcomes • Sayer (1984): Realism links theoretical abstraction with empirical research • “Essentially eclectic, the realist approach seeks to move away from what may be now rather sterile epistemological debates surrounding scientific versus pluralistic and constructivist paradigms, using all methodologies as appropriate within the realist framework. The approach facilitates the interrogation of empirical data with a view to identifying the nature of specific structural influences on outcomes” (p.45). • Realist evaluation embraces outcome measurement and also investigates the factors that influence the outcomes
Realist Evaluation • Realist evaluation seeks to evaluate practice within the realities of society, based on Emergent and Symbiotic elements of natural law. • All systems are emergent and constantly in a state of evolution, along with the reality that we are all symbiotically connected to nature and to each other. • Practice takes place in an open system that consists of a constellation of inter-connected structures, mechanisms and contexts. • Realism aims to address all the significant variables… through a realist effectiveness cycle which links the models of intervention with the circumstances in which practice takes place (Rom Harre’s models, analogous with reality) • Explanation at any one time requires further investigation and further explanation (continuous testing & development of Rom Harre’s models)
Purposes and Objectives of evaluation • Research methods drawn from both epidemiology and efficacy research traditions in a realist evaluation • Partnership with human service agencies & schools to investigate what works and for whom. Real live data mined from human service agencies is used to investigate the effectiveness of the human service interventions. • Emphasis is on data naturally drawn from practice. • Binary logistic regression as part of epidemiologic evidence based on association, environmental equivalence, and population equivalence. • How evaluators and agencies can make the best use of the available data to inform practice.
Purposes and Objectives of evaluation • Research indicates that the reform of the system of care and the use of initiatives such as wraparound are effective in improving mental health and functioning in school (Kutash, 2006; Reback, 2010; Goldenson, 2011) • However, most studies have focused on at risk groups rather than the total school populations. • The purpose of this study is to undertake a 100% evaluation of all school-based services, utilizing data on the entire school populations, in a longitudinal study.
Recommend
More recommend
