DPME Evaluations NQF Impact Workshop, Midrand Antonio Hercules (DPME) 09 February 2016
Outline 1. Evaluation : what is it? [2 minutes!] 2. NES and Types of Evaluation . what these are [6 mins!] 3. NQF Impact Study vs. NES Impact Evaluation . the study and impact evaluation [5 minutes!] 4. 2017/18 NES Implementation Evaluation current NES evaluation [5 minutes!] 5. Presentation Key messages 2
Evaluation – easy word, specific meaning As a field, its value and its shortcomings Field : 1. about 30-40 years, imprecise but improving, specifically deals with policy/programme/project as the object, specialised, mixed methods, sensitive to bias Value : 2. offers “objective” external and “expert” internal approach options. great window for renewal/improvement, highly flexible, decades of practice strongly evidence-based 3
Evaluation – easy word, specific meaning Shortcomings 3. Complex Duration Financial investment Must have buy-in Needs good and committed managers/political principals to take forward Evaluation is different from “normal” evaluative reflection in policy analysis/research 4
National Evaluation System and Evaluation Six (6) guidelines (for policy/prog./proj.) 1. Diagnostic *: … “identification” “research” – info. foundations 2. Design* : “formulation” 3. Implementation : performance (results), everything short of impact 4. Impact : using quantitative RCT approach conventionally 5. Economic *: …economic benefits 6. Synthesis *: …synthesis of pre-existing evaluations 5
NQF Impact Study vs. NES Impact Evaluation NQF Impact Study has value in its own right 1. Key question: would Study pass in the NES? 2. Main reasons: 3. Largely impact evaluation TOR, but methodology does not deliver accepted evaluation design More of a “synthesis” evaluation pulling together existing bodies of knowledge -> produces overview, trends, insights, but doesn’t subject underlying evaluation design of supporting reports to data verification/validation testing Does not appear to comply with accepted large system impact evaluation design (correlations, regressions, etc.) Still very useful, and well-timed for NQF Act policy implementation evaluation (NES – NEP 2016/2017) 6
Typical Impact Evaluation Approach • Literature • Theory of Change • Interviews • Policy 1 Results • Logframe 3 • Focus Frame- 2 Logic • Goals Evidence Groups • Operational work • Results Strategy • Survey • “Nesting” • Databases • Policy Narrative • Sound • Evidence 4 5 • Specific Assess- • Analysis Recom- • Based on ment • Results mend- evidence • Conclusions ations 7
2017/18 NES Implementation Evaluation Purpose: to assess the implementation of the NQF Act relative to its goal(s) and objectives in the period of review, including its associated policies and regulations. Successes and challenges in its implementation will be identified, and recommendations offered regarding improvements to implementation of the NQF Act in the future. 8
2017/18 NES Implementation Evaluation ( Relevance and Appropriateness ) To what extent is the 1. theory of change (intervention logic) of the NQF Act adequately robust, including its main underlying assumptions? ( Effectiveness ) To what extent has the implementation 2. of the NQF Act been effective in achieving its policy goal(s), objectives and intended outcomes? ( Efficiency ) To what extent has the implementation of 3. the NQF Act been efficient? ( Impact ) What is the emerging impact of the NQF, if any? 4. What needs to be done to improve the implementation 5. of the NQF Act, and which aspects of the NQF Act need to be reviewed ? 9
National S chool Nutrition Programme TOC Impact Activities Outputs Outcomes Inputs Immediate Intermediate Improved health Department of and nutritional Education pays Funds status of South for food delivered African primary school children Improved levels of Nutritious meals Nutritious Children have primary school delivered to meals adequate attendance schools on time eaten nutrition Schools have adequate Local Food prepared Improved food storage SMMEs locally and safely concentration facilities appointed to in class render services Local producers provide the Sufficient food/ingredients at teaching fair price time Local business Decentralised Fair prospers purchasing procurement Payments to system producers/suppliers Better made on time educational performance Assumptions 10
TOC example: Aspen Institute (2003) 12
Thanks! Antonio Hercules: Director: Evaluations & Research. antonio@presidency-dpme.gov.za Head: Evaluation and Research, DPME. Ian Goldman. Ian@presidency-dpme.gov.za www.thepresidency-dpme.gov.za
Recommend
More recommend
