BUILDING CAPACITY FOR EFFECTIVE USE OF POLICY EVALUATION: LESSONS - PowerPoint PPT Presentation

BUILDING CAPACITY FOR EFFECTIVE USE OF POLICY EVALUATION: LESSONS FROM THE OECD EXPERIENCE Jacobzone Stephane OECD, Public Governance Directorate 23 MAY 2019 1 8TH INTERNATIONAL EVALUATION CONFERENCE 1

WHY BUILDING CAPACITY FOR POLICY EVALUATION? Evaluations have a critical role to play in improving the quality, responsiveness and accessibility of public services . › It can play a role throughout the policy cycle › It helps to prevent one-sided policy design, avoid duplication and ensure scarce resources are well used › It helps with policy implementation and adapting policies to meet local needs › Policy evaluation is also critical to understand why policies do or don’t work 23 MAY 2019 2 8TH INTERNATIONAL EVALUATION CONFERENCE 2

WHY IS IT SO DIFFICULT FOR EVALUATIONS TO BE USED ? Ensuring demand for evidence and evaluation has become very challenging in a context of global over-supply of knowledge and complex political process. › Information to be considered by policy makers is overwhelming and complex › Evidence gaps remain on ‘what works’ in many policy areas › The challenges of navigating the ‘post truth’ world › The rise of wicked problems › The erosion of trust in public institutions › The challenge to the authority of science 23 MAY 2019 2 8TH INTERNATIONAL EVALUATION CONFERENCE 3

Evaluation is part of good public governance and performance management frameworks Performance Spending Review Budgeting Evaluation Oversight and HRM citizen engagement

PRELIMINARY RESULTS OF THE OECD CROSS COUNTRY SURVEY ON POLICY EVALUATION On going survey (42 countries to date) The objectives are to understand:  Key objectives of evaluation and how countries mobilise evaluation towards these objectives  What are the evaluation practices  Why countries are conducting policy evaluation, what challenges they encounter and what good practice they identify Note : the data presented here are not for any public diffusion and are still undergoing a consultation and validation process with participating countries. Complemented with some results from the survey on Budgeting and Results 5

What elements contribute most to the definition of evaluation ? While differences exist, there are some recurring elements across countries • The definition used by the OECD % of countries using the following elements of their in the questionnaire : definition of policy evaluation Policy evaluation is understood as Ex-ante Initiative the structured and objective Reasoned Neutral assessment of an ongoing or Ex-post completed policy or reform initiative, Analysis RIA its design, implementation and Process Evidence results. The aim is to determine the Regulation Objective relevance and fulfilment of Intervention objectives, efficiency, effectiveness, Criteria Efficiency impact and sustainability, etc. Effectiveness Impact Evaluation also refers to the process Programs of determining the worth or Regular Avalaible significance of a policy . Rigorous Used Policy Systematic 0.0 10.0 20.0 30.0 40.0 50.0 * The grey elements correspond to the evaluation base of 6 Lazaro (2015).

UNDERSTANDING POLICY EVALUATION SYSTEMS Implementing a holistic national evaluation systems contributes to good public governance across the policy cycle. The OECD survey analyses the maturity of evaluation systems across 3 criteria: • The Institutional Framework of evaluations offers (a) the legal base to Institutional perform policy evaluations (b) provides a Organisation i macro orientation as to when and how to perform policy evaluation; (c) identifies and gives mandates to institutional actors with corresponding resources for supervising, controlling and performing policy evaluations Promoting Promoting use quality of of evaluation e valuation • An evaluation driven culture , which includes efforts towards promoting quality and use of evaluations across government, for example through training, investing in skills and stakeholder engagement 7

What are the perceived challenges ? Use of evaluation results in policy making Strategy for policy evaluation promoting a whole-of- government approach Quality of evidence Human resources (capacities and capabilities) for policy evaluation Political interest in, and demand for, policy evaluation Average Strong mandate of the main institution responsible for policy evaluation Financial resources for carrying out specific policy evaluations Adequate legal framework for policy evaluation Financial resources of the main institution responsible for policy evaluation 0 1 2 3 4 5 6 7

Evaluation systems : Defining an institutional framework • The institutional frameworks include: o Actors and mandates : within the public sector (within executive, or beyond (e.g. SAIs), or on the valuation market (NGOs, Think tanks, civil society) o Normative frameworks (laws, regulations, strategic orientations, etc.) • The OECD questionnaire focuses on public institutions and the executive sphere in general. Other institutions, such as SAIs are reviewed in terms of their relation to the executive. 9

THE GOVERNANCE OF EVALUATION CBA 30 25 Line Ministries / Agencies 20 15 10 5 0 External experts (e.g. Legislature consulting firm or university) Government evaluation Supreme Audit Institution service(s) ex ante ex post Source: OECD survey on budgeting and results

Legal and regulatory instruments for policy evaluation Secondary/subordinate legislation Primary legislation (law/s or equivalent) Constitution 0 5 10 15 20 25 11

Key elements of the policy frameworks Responsibilities of government institutions concerning policy evaluation Policy areas (thematic) or programmes covered by the evaluation policy Objectives or expected results of the evaluation policy Requirements related to evaluation reporting Requirement for government institutions to undertake regular evaluation of their policies Requirements related to stakeholder engagement Requirements related to the use of evaluation findings into policy planning making Requirements related to the quality standards of evaluations Standards for ethical conduct 0 2 4 6 8 10 12 14 12

Key parts within the executive with competence for policy evaluation Centre of government and Ministries of Finance Centre of Government / Presidency / Prime Minister’s Office / Cabinet Office or equivalent Other Ministry of Public Sector Reform / Modernisation / Public Function or equivalent Autonomous Agency Competences for policy evaluation are not explicitly allocated to specific institutions 0 5 10 15 20 25 30 13

Key component of the institutional frameworks beyond the executive SAI have overwhelming competences in a large majority of countries Supreme Audit Institutions or similar Congress/Parliamentary Budget Offices Other/ Please specify underneath None of the above 14 0 5 10 15 20 25 30 35 40

Promoting quality and use of policy evaluation • A solid institutional framework and clear orientations for policy evaluation will lack impact if staff are not equipped with the right skills and exposed to the right incentives • In particular : o Quality of evaluation allows to determine if the data that are produced represent trusted evidence, or if they can facilitate learning and accountability by public officials, contributing to improved decision making and policy design o Use of evaluations is indispensable to the extent that :  These require significant public resources  If they are not used, the data are also likely to suffer Note : OECD is also assessed in terms of the quality and impact of its work 15

Promoting quality of evaluations • Quality of evaluations can be promoted through : o Mechanisms to control the quality of deliverables o Mechanisms to control the process o Other mechanisms: requirements in terms of competences, legal frameworks, role of SAIs • Quality of evaluation can also be promoted through: o Interpersonal mechanisms: stakeholder engagement. o Systematic approaches :  Content (quality, communication, etc.)  Investment in skills  Context (policies, institutions, budget calendar, etc.) 6

How can countries promote quality ? quality assurance and external control Countries are using various mechanisms for quality assurance and control 40% 35% 30% 25% 20% 15% 10% 5% 0% Provisions expressed in a Competence requirements for Peer review (internal/external) of Systematic and meta-evaluations Other, please specify underneath There are no mechanisms to ensure policy/legal framework evaluators evaluations (The term is used for evaluations the quality of evaluations across designed to aggregate findings from government a series of evaluations. It can also be used to denote the evaluation of an evaluation to judge its quality and/or assess the performance of the evaluato EXAMPLES: • A country can foresee that all evaluations are subject to peer review through a joint committee of ministries’ experts, practitioners and representatives of civil society; • The European Commission has developed a map of competences that are necessary to increase the quality of 17 evaluations