2020 Application of data analysis in public auditing of the national goals attainability: evidence-based policies in contemporary state governance
1. Theory of change and theory of transformation: application for analysis of national goals attainability
Theory of change: evaluation of federal projects Theory of change is a visualization of causal steps: Outcome STEP … Intervention THEN THEN THEN (intended) 1 ToC includes: ToC is used: • on the planning stage: Assumptions : Risks : to justify the logic of the intervention. • • for the monitoring: events and the inverse of • to analyze intermediate results and to adjust conditions that assumptions; if the intervention. need to occur for they occur, the • for the evaluation: the causal link to casual link does not realize; hold; • to assess the final outcomes of the policy. Source: Befani, B., & Mayne, J. (2014). Process Tracing and Contribution Analysis: A Combined Approach to Generative Causal Inference for Impact Evaluation. IDS Bulletin, 45(6), 17–36.
Assumptions Chain process of changes Steps Metrics Sources Activities Distinct events of the Federal Data on the monitoring of the Federal The preschool Reports on the Project’ objectives fullfillment prescribed by Project “Assistance for women Projects institutions have really the policy with children” been created 1. Number of places in preschool Data on the monitoring of the Federal institutions Preschool facilities with Policy output The preschool 2. Amount of money subsidized to the Projects predetermined number of places institutions are private preschools are created logistically considerate The price is reasonable People send their children to the or free of charge preschool institutions 1. Number of children under 3 years visiting Intermediate 1. Data on the monitoring of the Women prefer to send preschool institutions outcomes Federal Projects children to the Mothers obtain more free time 2. Active employment among women with 2. Russian statistical service data institutions rather than which could be allocated to the children under 3 years 3. Data from the Ministry of Labor take care themselves work The social payments Mothers are employed and get are still paid after paid employment No discrimination towards women with Increase in incomes, decrease in kids on the labor force market poverty Final outcomes Income in households with women having children The policy affects poor households
2. Evidence gap maps: Facilitating strategic accumulation of evidence
Evidence Gap Maps Visualization of existing evidence of effects of interventions WHAT IS EGM? Existing studies MOTIVATION • have uneven quality • require time to find and extract valuable evidence – great number of different sources, studies are large in size • support for evidence-based evaluations of interventions RESULT • easy access to the information about the number of related studies, level of their quality and main findings • identification of existing evidence gaps • facilitation of future research aimed to fill existing evidence gaps Source: Snilstveit, B., Vojtkova, M., Bhavsar, A., Stevenson, J., & Gaarder, M. (2016). Evidence & Gap Maps: A tool for promoting evidence informed policy and strategic research agendas. Journal of Clinical Epidemiology, 79, 120–129.
Evidence Gap Map: example Medium quality Low quality High quality Outcomes State interventions Household income growth Decreasing poverty rate Decreasing unemployment 1. Creating new preschool education facilities Dujardin et al. (2018) Karabchuk, 2013 2. Social contracts to help job search and employment Biewen, M. and Blundell and J. van Waller M. (2007) Reenen (2004) 3. <…> <…> <…> <…>
Assessment of evidence quality 1 2 3 4 5 6 7 Very low quality of Final score 1 - 1,5 evidence № Criteria Very low Low Medium High Very high Criterion Calculation quality quality quality quality quality weight Low quality of Final score 1,5 -2,5 evidence 1 Study type 4 0.8 3.2 Medium quality of Final score 2,5 - 3,5 Country-specific 2 2 1.4 2.8 evidence compliance High quality of Relevance (time Final score 3,5 - 4,5 3 1 0.6 0.6 evidence context) Very high quality of 4 Causality 3 1 3 Final score 4,5 - 5 evidence 5 Reproducibility 5 0.6 3 0 1 - ∗ 3 - !"#$% &'()* = , -./ Conformity of 6 3 1.6 4.8 mechanism 5 Final score 2.90 1 - – evidence score for the criterion " 3 - – weight of criterion "
Closing Evidence Gaps US: President Obama’s evidence-based UK: What Works Network initiatives (2016) • collating existing evidence on the Building and Using Evidence to • effectiveness of programs and Improve Results practices Investing in Child Care and Early • • producing high quality synthesis Learning : support universal reports and systematic reviews in preschool through the Preschool for new areas All Initiative • assessing the effectiveness of Fund early education research and • policies and practices against an evaluation by supporting the agreed set of outcomes Institute of Education Sciences’ • filling gaps by commissioning new Early Learning Research Network trials and evaluations • sharing findings across institutions in an accessible way
Challenges in evidence-based policy evaluation No relevant studies Existing studies have poor design Existing studies are based on unlikely assumptions Existing studies do not comprehend Russian peculiarities
Challenges in evidence-based policy evaluation: data Unsuitable format Data processing methods change over time No panel dimension Representativity problems Poor regional presence in data
3. Policy evaluation: case of healthcare reforms evaluation
Healthcare dynamics Source: Federal state statistics service
Evaluation of healthcare spending on mortality, models Model I: pooled OLS %&'()*+(, -" = / + 1! -" + 2# -" + 3 -" Model II: panel with trend %&'()*+(, -" = / + 1! -" + 2# -" + δ$ -" + 3 -" %&'()*+(, -" = / + θ%&'()*+(, -"67 + 1! -" + 2# -" + δ$ -" + 3 -" Model III: panel with trend and lag %&'()*+(, -" = / + 1! -" + 2# -" + δ$ -" + κ! -" $ -" + 3 -" Model IV: panel with interactions %&'()*+(, -" Model V: panel with trend and lag = / + θ%&'()*+(, -"67 + 1! -" + 2# -" + δ$ -" + κ! -" $ -" + 3 -" ! " – healthcare spending # " – regional control variables* $ " – year dummies 14 *income, unemployment, poverty, inflation, food consumption by categories, ecology by categories, age composition, crime rate
Results Source: Own calculation
Oaxaca-Blinder decomposition for policy evaluation Is it possible to use such a decomposition to uncover effects of certain policy over time?
Decomposition: any structural effect? Source: Own calculations
Contact information Daria Tsyplakova, Director of Department for research and method@ach.gov.ru methodology, Accounts Chamber of the Russian Federation Yulia Filippova, Senior Advisor, Accounts Chamber of the Russian Filippova_YA@ach.gov.ru Federation Egor Zhevlenev, Analyst, Accounts Chamber of the Russian e.zhevlenev@cpur.ru Federation
Recommend
More recommend
