The Evaluation Center Western Michigan University The Organizational and Political Landscape for Evidence-Informed Decision Making in Government Kathryn Newcomer November 30, 2016 1
“Evidence-based Policy,” “Data-Driven Decision-making”– the New Normal? 2
Questions to Address Today u When and why did the “evidence-based policy” imperative become so prevalent in the public and nonprofit sectors? u How can evaluators help government decision- makers use evidence to inform decision-making? u How can we move from generating data for accountability to learning ? 3
“Evidence-based Policy” u The evaluation illuminary affecting governmental decision-makers, foundations, nonprofit boards, intermediaries and --- evaluation practice! u Myth or reality? u Advantages and disadvantages for decision- makers and for evaluators? 4
Efforts in Several Arenas Have Moved the Dialogue to Embrace “Evidence–Based Policy” u Since the 1960s in the U.S. dialogue about the target for government’s efforts has changed from a focus on effectiveness to outcomes to results to evidence ---- Why? u Tracking of diseases in Public Health, e.g., “Healthy People 2000,” and The Cochran Collaboration u North American and European Social Scientists Established the Campbell Collaboration to mimic The Cochran Collaboration u The U.S. Office of Management and Budget’s efforts to Assess “evidence of program effectiveness” in both the George W. Bush and Barrack Obama Administrations were highly influenced by the Coalition for Evidence-based Policy u Leading Foundations have invested resources to encourage evidence- based decision-making, e.g., Pew, MacArthur, Arnold, and Grant u Evaluation and Monitoring of International Development Efforts 5
From Outputs to Evidence: Influential Events Across the Years Coalition Some Federal Evidence- Hatry Senate laws require OMB Guidance on Tiers of Healthy Call for Based policy Report on outcome Evidence (2010) People Key gains traction measuring measures Simon & “2000” National + PART (2001 ) effectiveness (1970s ) Ridley focus (1990) indicators (1967 ) United Way on outputs Urban (2004) GASB calls CompStat requires (1938) Institute & for Service focuses on Outcomes IMCA Oregon Accomplish Crime Rates in Measures (1996) Gates defines Performance Benchmarks State & Local ments data NYC (1992) What results as both Work Begins (1989) Finance (1980) Works output and Millennial (1970s) Project Community clearing outcomes challenge Indicators (1960s) Cochran house (2013) sets Consortium Collaboration established impact established is established (2002 ) goals (2004) (1993) (2000 ) 2010s 1930s 1960s 1970s 1990s 1980s 2000s CDC GPRA (1993) Promotes OMB Guidance Workforce training laws World Bank DEBIs on Evidence- Focus on requires outcome calls for (2004) Based Grants effective-ness measures outcome (2010) within DOD & (1982) measures HEW (1963) CHEA establishes Moneyball (1990s ) Outcomes Standards published (2003) CNCS Social (1998) Mental Health Innovation Outcomes Fund (2009) Pew- Measured (1970s) MacArthur Reinventing Results First Hitch and Campbell Collaboration Government (2011) McKean (1960) established (2000 ) published (1993)
Embracing Evidence-Based Policy OMB Cochran Guidance on Coalition for Collaboration Tiers of Evidence-Based Health is established Evidence Call for Key policy gains People (1993) Moneyball for (2010) National traction “2000” Government indicators (2001) (1990) published (2004) United Way requires (2014) Outcomes Assessment (1996) CompStat What Works Community focuses on clearing Indicators Pew- Crime Rates Millennial house Consortium MacArthur in NYC challenge sets established established Results First (1992) impact goals (2002) (2004) (2011) (2000) 1990s 2010s 2000s Pew- CDC World Bank calls MacArthur Promotes OMB for evaluation of GPRA Results DEBIs Guidance on outcomes (1990s) (1993) First (2004) Evidence- initiative Based Grants CHEA (2001) (2012) establishes Outcomes CNCS Social Standards Innovation (1998) Michael Osborne and Congress Fund (2009) Lewis’ book Gaebler book Campbell Internationa Passes the Moneyball Reinventing Collaboration Initiative Commission published Government established for Impact on (2003) published Evaluation (2000) Evidence- (3ie) (1993) Based (2008) Policy Act (2016)
Evidence-Based Policy – Made by Whom? Decisions to be Informed by Evidence Basing funding on use of Political “Demonstrated Evidence-Based Interventions” (DEBIs) and/or CEA Making programmatic decisions Programmatic based on impact evaluations Analyzing programmatic data – Operational preferably outcomes – to target resources 8
Contrasting Views on Evidence-Based Policy Fixed Mindset Growth Mindset 1. We need to collect data to test if programs 1. We need to learn which program mechanisms work or do not work. work for whom, where and under what circumstances. 2. Policy should be made at the top and based on 2. Policy is “made” through implementation evidence. processes at multiple levels by multiple actors with different types of data available to them. 3. Program impact can be measured precisely. 3. Measuring program impact is difficult as programs and intended impactees change and evolve. 4. Random Control Trials (RCTs) are the gold 4. Research designs must be matched to answer standard for research and evaluation design. the question raised; RCTs are appropriate for certain impact questions. 5. Proven program models can be replicated in 5. Program mechanisms may be replicated in multiple locations as long as they are implemented multiple locations as long as they are adapted to with fidelity to the original design. meet local conditions. 6. Benefit-cost analysis should be used to compare 6. Benefit-cost analysis is difficult to use to social programs. compare social programs given the challenge of costing out benefits , especially those accruing over time. 9 Note: I expanded upon the notion of mindset in Mindset by Carol Dweck.
Obama Administration: Explicit Emphasis on Producing and Acting on Evidence A series of office Memoranda from OMB between 2009 and 2013 signaled that u performance measurement and evaluation were to be used to produce "evidence on what works" Starting in 2015 OMB Circular A-11 defines evidence for the federal u government: u “For purpose of A-11 Part 6, evidence is the available body of facts or information indicating whether a belief or proposition is true or valid. Evidence can be quantitative or qualitative and may come from a variety of sources, including performance measurement, evaluations, statistical series, retrospective reviews and other data analytics and research. Evidence has varying degrees of credibility, and the strongest evidence generally comes from a portfolio of high-quality evidence rather than a single study.” 10
The Obama Administration Recognizes Tiers of Evidence Based on Perceived Rigour Preliminary/Exploratory Moderate/Suggestive Strong/Causal Evidence Evidence Evidence multi-site experimental evaluations grounded on theory, participant pilots, experimental tests, single- of standardized approach, PLUS tracking, evaluability site experimental evaluations, non- structured implementation analysis assessment, structured case experimental statistical modeling, and optional ethnographies and studies, documentary performance analysis, structured statistical modeling implementation studies, implementation analyses/ developmental evaluations evaluations, formal ethnographies
What are Challenges for Evidence to Inform Policymaking? u Expectations regarding: u What constitutes evidence? u How transferable is evidence? u When and where do we underestimate the role played by the “impactees?” u Where is the capacity to support both the demand and supply of evidence? 12
We Overstate the Certainty of the Evidence we Can Collect Ø Perceptions of the certainty of “evidence” have changed. 13
What are the Opportunities for Evidence to Inform Decision-making? u Analyses of “performance” data collected by agencies (or delegated service delivery agents such as grantees) u Implementation, Outcome and Impact evaluations typically performed by other agents for government u Manipulations of services in experiments by agencies – “behavioral economics” u Syntheses or systematic reviews of impact evaluations by external agents, e.g. websites like “What Works” 14
Why isn’t There Agreement About the Quality of Evidence? u Differing professional standards and “rules” or criteria for evidence, e.g., lawyers, accountants, engineers, economists u Disagreements about methodologies within professional groups, e.g., RCTs u The constancy of change in problems and the characteristics of the targeted impactees 15
“Evidence-Based” Grant Making u Grants comprise over $600 billion in the US federal Budget u OMB started urging agencies to use evidence based grant making starting in 2010 but with little guidance u Where are we now?
Recommend
More recommend
