Evaluation Talk From Uganda, Ukraine and beyond – Evaluation lessons from parliamentary strengthening? A practical example of the design and execution of a multi-country end-evaluation By: Karin Weber Kampala, April 9 th 2014
Presentation Outline 1. Introduction 2. Scope of evaluation 3. TWC evaluation in numbers 4. Purpose of evaluation 5. Evaluation criteria 6. Evaluation methodology 7. Evaluation framework 8. Limitations & Challenges 9. Top tips 2
Introducing… Karin Weber - Senior Consultant - M&E specialist - Worked as independent consultant, NGO director, United Nations M&E specialist, MoFA policy advisor in Europe, Africa and Asia. Delta Partnership - International development consultancy firm - Offices in UK, Kenya and Uganda - Recently became part of WYG - Multi-sector and multi-disciplinary
TWC end evaluation in numbers… 7 Consortium members 1 lead 4 thematic 7 Countries in areas 5 different 5 outputs regions TWC end evaluation 130+ 50 respondents Implementing partners 5 languages 9 Evaluation Criteria
Evaluation Scope (1/3) - The programme: ‘’Strengthening Human Resource Development in Southern Parliaments’’ - Implemented by The Westminster Consortium (TWC): - led by the Westminster Foundation for Democracy (WFD) - Commonwealth Parliamentary Association UK Branch, - The House of Commons Overseas Office, - The International Bar Association’s Human Rights Institute - The National Audit Office (NAO) - The Thomson Reuters Foundation (TRF) - The University of Essex Institute for Democracy and Conflict Resolution - Funded by DFID’s Governance and Transparency Fund (GTF)
Evaluation Scope (2/3) • TWC programme aimed to establish self- sustaining Parliamentary Studies Centres (PSC) and building the capacity of local partners to deliver effective parliamentary development • Twinning model between UK partners and key partners in each country. • Four thematic focus areas: 1. Parliamentary Process and Management; 2. Human Rights and the Rule of Law; 3. Financial Oversight; and 4. Access to Information.
Evaluation Scope (3/3) • Period under evaluation: 2008 – 2013 • Geographical scope: UK, Georgia, Ukraine, Uganda, Mozambique, Lebanon and Morocco
Purpose of Evaluation In line with GTF guidelines the purpose of the Final Evaluation was to: Identify the impact of the programme and ways that this may be sustained Record and share lessons Account to local stakeholders for the programme’s achievements Improve future programme design and management Verify funds were used effectively and efficiently to deliver results Enable DFID to evaluate the performance of the GTF as a whole, making sure the overall portfolio has increased accountability and responsiveness.
Evaluation Criteria • Relevance • Impact • Economy • Efficiency • Effectiveness • Equity • Value for money • Sustainability • Replicability
Methodology • The international evaluation team: Karin Weber and Aileen Lyon • The end evaluation was conducted between September 2012 and August 2013 • Step 1 : Planning meetings with the client • Step 2 : Stakeholder mapping (to identify potential respondents) • Step 3 : Development of an evaluation matrix • Step 4 : Development of an evaluation plan • Step 5 : Development of interview topic list/ questionnaires • Step 6: Conducting the desk study • Step 7 : Carrying out 6 country visits (up to 5 days in each country) – Mid-term update report to client including 3 visit reports and emerging findings – Country reports validated with in-country teams • Step 8 : Analysis and draft reporting • Step 9 : Validation meeting with TWC team • Step 10 : Finalisation of evaluation report
Example of Evaluation Framework Criteria Key evaluation questions Methodology Data sources/key stakeholder • Effectiveness To what extend have the planned Desk study Project results been achieved to date? documents • What has been the most Semi- significance change of the structured National programme? interviews level studies • Which factors contributed to the and surveys success of the programme? Focus Group • What were the key challenges Discussions Project staff, affecting programme performance? Ministerial • Were there any unintended results policy in the programme? Were they advisors positive or negative and in which ways did they affect the different stakeholders?
Cont. Criteria Key evaluation questions Methodology Data sources/key stakeholder • Efficiency How economically were Value for Project resources/inputs (funds, expertise, Money documents time, etc.) converted to programme analysis results? Finance • To what extend did the budget and Cost-benefit officers actual expenditure at the country analysis level reflect identified programme Audit priorities? Desk study reports • Relevance Was the programme formulated Interviews Project and implemented in accordance document with national and local strategies? Desk study • To what extend did the programme National answer the needs of the intended Satisfaction policies beneficiaries? survey Beneficiaries
From ToR to Framework, Tools and Analysis Evaluation objectives In Terms of reference Link to Evaluation criteria Evaluation framework Evaluation Criteria & Evaluation Questions Methodology & Data sources & Key stakeholders Tools for data collection Interview topic lists & Questionnaires Desk review checklists & Achievement rating scale Analysis and Reporting Labelling, grouping, summarising Report outline/ chapters
Desk Study Other documents TWC Programme Evaluation documents • Final Evaluation Guidelines for documents Governance and Transparency • Log frame Fund Grant Holders • Mid Term Review • Budget • National level data (incl. bills, Report laws, policies, statistics etc.) • Quarterly reports • Management • Academic literature related to • Annual reports response to MTR topic • Financial reports • Internal • Articles by trained journalists evaluations • Risk management • Handbooks and guides report developed by the programme • Partner documents • Case studies • Websites
Methodology (cont.) • Country visits: – Observation, Semi-structured interviews and Focus Group Discussions with: • TWC Programme Managers and Administration and Finance Officers • local partners • MPs and Parliamentary staff • Local trainers and Trainees • Journalists and representatives from Civil Society Organisations (CSO). • DFID and other donors working with the parliament in-country – Use of a translator in Georgia, Ukraine, Morocco, Lebanon and Mozambique – Use of voice recorder where permission was granted by respondent – Additional in-country document review where relevant • Performance analysis - to assess performance against the logframe, for outputs, outcome and impact
Methodology (cont.) • Data analysis and synthesis – Notes from document review – Transcriptions from interviews and FGDs – Labelling and synthesis in line with evaluation matrix – Further analysis of financial data to assess Value for Money • Reporting – Draft report in line with good practice, the ToR and GTF guidance – Discussion and validation with the TWC team – Incorporation of feedback – Finalisation of the evaluation report.
Challenges & Limitations (1/2) • Limited evidence – Especially at the higher purpose (outcome) and goal (impact) levels. – Focus on monitoring activities and expenditure during implementation instead of intermediate and long-term results – Limited use of the logframe for monitoring (seen in-country as a HQ document) – Poor follow-up after trainings, study tours and other capacity building activities – No systematic monitoring of bills passed and implementation of laws – Lack of clarity on the responsibility for data collection during monitoring • Data collection during country missions took place at different times in the last programme year (between October 2012 and June 2013)
Challenges & Limitations (2/2) • In some cases WFD staff members were present during interviews; risk of bias • Security (e.g. Lebanon mission postponed due to car bombing next to Parliament) • Political sensitivity (e.g. parliamentary proceedings in Lebanon are not public) • The end evaluation steering committee consisted of TWC members only; no independent members • Gender unbalanced evaluation team • Different quality of translators
Top Tips (1/2) • Check specific donor guidelines for end evaluations • Frequently refer back to ToR and evaluation matrix • Ensure a good preparation and initial analysis before the country missions: – Go straight to the monitoring data and reports early on to get a grip on the data they have available – Ensure that you have a ToC up front – not just the logframe. Jointly develop one if ToC doesn’t exist. You can then test the assumptions and how it all worked out as you go into in-country conversations • Check country reports, data and facts with respondents and/or managers before the final analysis of the data • Make time for a joint analysis with all the team who worked on it – much more efficient
Recommend
More recommend
