Presentation on Impact Measurement Concern Worldwide - 22 nd July - PowerPoint PPT Presentation

Presentation on Impact Measurement Concern Worldwide - 22 nd July 2008 Michael King Trinity College Dublin IIMDA - Initiative on Impact Measurement in Development Aid

Outline 1. Establishment of IIMDA 2. Overview of Impact Measurement Methodologies 3. Case Study 1: Electoral Violence in Nigeria IIMDA - Initiative on Impact Measurement in Development Aid

1. IIMDA • Trinity College/IIIS Initiative led by Pedro Vicente and Michael King with research partners in UCD, Oxford University, Columbia University and Tufts University. • IIMDA Mission – Research: To develop understanding on the impact of innovative development aid projects. – Dissemination: To communicate lessons learned from impact evaluations to policy makers and development practitioners. – Training: To develop the impact evaluation skills of development practitioners and researchers based in Ireland and partner countries. Speakers at the IIMDA Conference on Impact Evaluation (April 17 th 2009) IIMDA - Initiative on Impact Measurement in Development Aid

IIMDA Developments 2009 • Ongoing project: Counteracting Illegal Behaviour during Elections in Mozambique: The Role of Observers and Cell Phones (in partnership with Oxford and Tufts Universities). • Forthcoming Projects: Agreed to develop joint research projects with Trocáire, Christian Aid Ireland and Microfinance Opportunities in Washington DC. • Wish to explore the demand for intensive training courses in impact evaluation. IIMDA - Initiative on Impact Measurement in Development Aid

ABIA Recommendations and IIMDA Source: ABIA ((2007) Measuring Impact: The Global and Irish Aid Programme Research Training Dissemination Ensure that comprehensive assessments of aid impact encompass both ex ante and systemic as well as ex 1 post assessments. Judiciously expand the number, scope and range of 2 assessments of discrete aid projects. Undertake more systematic and in-depth sectoral and 3 country-wide assessments of aid Take stock of the relationships between aid and the 4 achievement of the MDGs Ensure aid assessment includes analysis of policy 5 coherence Extend, deepen and improve assessments of humanitarian 6 aid. Revisit the historical distinction between humanitarian and 7 development aid Support efforts to expand and extend NGO assessments of 8 impact Support more innovative activities and ensure that lessons 9 are learned IIMDA - Initiative on Impact Measurement in Development Aid

2. Impact Measurement (Evaluation) • Impact evaluation studies estimate the effect of an intervention on final welfare outcomes, rather than assessing the project outputs or the project implementation process. IIMDA - Initiative on Impact Measurement in Development Aid

Evaluation and Monitoring Framework Methodologies Welfare effects on target Impact Quantitative Outcomes group directly or partly Evaluation (Impact) attributable to the project. Qualitative Goods and services Outputs produced by the project Quantitative Implementation Actions and tasks carried out to Activities transform inputs into outputs Qualitative Monitoring Financial, human and material Inputs resources required Source: Adopted from Asian Development Bank (2006) Impact Evaluation: Methodological and Operational Issues. IIMDA - Initiative on Impact Measurement in Development Aid

Log Frame • The logical framework (“log frame”) considers the higher-order project results (“outcomes” and “impacts”) but alone lacks the tools to estimate robustly project impact. • In practice, the log frame is generally used to assess the operational flow of project inputs and outputs and assessing the validity of the underlying assumptions. IIMDA - Initiative on Impact Measurement in Development Aid

Log Frame and Impact Evaluation Methods IIMDA - Initiative on Impact Measurement in Development Aid

Alternative Impact Evaluation Methodologies IIMDA - Initiative on Impact Measurement in Development Aid

Random Assignment IIMDA - Initiative on Impact Measurement in Development Aid

Types of Randomization Mechanisms 1. Lottery design: Applicants are simply randomly assigned to the treatment group and the control group. – Used when the program resources can cover only a fraction of eligible participants and there is no reason to discriminate among applicants. – Perceived as a fair and transparent means to decide who will receive the program benefits and who will not. IIMDA - Initiative on Impact Measurement in Development Aid

Types of Randomization Mechanisms 2. Phase-in design can be used when the program is designed to cover the entire eligible population but in a phased in manner. – Everyone is told that they will end up receiving the program benefits but at different times. – The timing of actually receiving the program benefits can be randomized. IIMDA - Initiative on Impact Measurement in Development Aid

Types of Randomization Mechanisms 3. Encouragement design is used when everyone is immediately eligible to receive the program benefits and there is enough funding to cover the entire eligible population, but not everyone will necessarily take advantage of the program. – Can randomly select a group of people and offer them specific incentives to encourage them to use the program. – The remaining population without the incentives is used as the control group. IIMDA - Initiative on Impact Measurement in Development Aid

Issues in RCTs • For RCTs to be successful treatment and control groups must remain clean and unchanged as originally designed throughout the study period. The following three issues require specific attention. – Attrition: This is the situation when some members of the treatment or control group or both, drop out from the sample. – Spillover: This occurs when the program impact is not confined to program participants. – Noncompliance: This is another complication in randomized evaluation. It occurs when some members of the treatment group do not get treated or get treated improperly, or some members of the control group get treated. IIMDA - Initiative on Impact Measurement in Development Aid

Double Difference IIMDA - Initiative on Impact Measurement in Development Aid

Case Study • Votes and Violence: Evidence from an Informational Campaign in Nigeria (Pedro Vicente and Paul Collier) • Evaluation of the impact of a randomised anti-violence grassroots campaign. • Conducted by ActionAid International Nigeria IIMDA - Initiative on Impact Measurement in Development Aid

Project Background • Political violence in African elections In April and May 2003, at least one hundred people were killed and many more injured during federal and state elections in Nigeria. In a number of locations, elections simply did not take place as groups of armed thugs linked to political parties and candidates intimidated and threatened voters in order to falsify results. – June 2004, Human Rights Watch IIMDA - Initiative on Impact Measurement in Development Aid

Research Questions 1. Can a NGO-conducted campaign against electoral violence help in undermining this phenomenon? 2. Does violence diminish voter turnout? 3. Who are the candidates that are perceived as ‘violence-prone’? 4. Who are the people with whom the campaign works particularly well? 5. Are there network effects of the campaign against electoral violence? IIMDA - Initiative on Impact Measurement in Development Aid

Campaign Details 1. Distribution of materials • T-shirts (3,000, i.e. 1 for every 2 households on average), Caps (3,000), Hijabs (1,000) • Posters (3,000), Stickers (3,000), Leaflets (5,000) 2. Road shows (using jingles in 3 languages, Yoruba, Hausa, Pidgin English) 3. At least one Community Meeting at each of the 12 locations 4. At least one Popular Theatre representation at each of the 12 treated locations 5. Opportunity to report violence on election day. IIMDA - Initiative on Impact Measurement in Development Aid

Campaign Timeline Figure 5: The Time Frame of the Experiment Jan/Feb: Feb: May/Jun: Pre-Election Anti- April: Post-Election Violence Survey Survey Past Year Elections Campaign asking about violence asking about 1200 Respondents violence (96% Re-surveyed) Time Line Independent Journalists in all 24 EAs IIMDA - Initiative on Impact Measurement in Development Aid

Campaign Literature IIMDA - Initiative on Impact Measurement in Development Aid

IIMDA - Initiative on Impact Measurement in Development Aid

Popular Theatres IIMDA - Initiative on Impact Measurement in Development Aid

Anti-violence Roadshows IIMDA - Initiative on Impact Measurement in Development Aid

Sampling Process 1. Focused on AfroBarometer’s sampled EAs 2. Treatment areas chosen randomly 3. Non-random component: 2 states in each of the main 3 regions of the country (Southwest, Southeast, North) chosen by looking at recent history of violence 4. Control EAs chosen by choosing the closest EAs from AB’s list, in the same state, of the same type (Large Urban, Small Urban, Rural) 5. Households and corresponding respondents within a census area chosen randomly using AB’s standard techniques 6. Conditions for sampling within a household: 18+ years, residence in EA IIMDA - Initiative on Impact Measurement in Development Aid

IIMDA - Initiative on Impact Measurement in Development Aid