2012 AFDB Evaluation Week: Evaluation for Development IMPACT EVALUATION: LESSONS FROM AFRICA Annette N. Brown www.3ieimpact.org
Outline of talk • What is impact evaluation • Colin Powell’s rules redux: rules for good impact evaluation studies • Evidence I: Institutions • Evidence II: Roads • Evidence III: Sustainability Annette N. Brown www.3ieimpact.org
Impact Evaluation An evaluation that examines the counterfactual in order to attribute changes in outcomes to an intervention. Annette N. Brown www.3ieimpact.org
Good Impact Evaluation A good impact evaluation explores all the links in the causal chain. This involves using mixed methods. Not quantitative and qualitative, but counterfactual and factual. Annette N. Brown www.3ieimpact.org
Mixed methods Howard White, “The use of mixed methods in randomized control trials”, 3ie 2012. Annette N. Brown www.3ieimpact.org
Colin Powell’s rules • Tell me what you know • Tell me what you don’t know • Then tell me what you think • Always distinguish which is which Annette N. Brown www.3ieimpact.org
Good impact evaluation rules Colin Powell’s rules redux • Tell me the counterfactual • Tell me the factual • Tell me what you don’t know • Then tell me what you think • Always distinguish which is which Annette N. Brown www.3ieimpact.org
Evidence I: Institutions “Effective conflict resolution is essential to order and development. Every local business deal, land boundary, or loan offers a possible conflict. When local dispute institutions function well, they prevent and resolve conflict.” (p. 1) “Building institutions at the micro -level: Results from a field experiment in property dispute and conflict resolution” Christopher Blattman, Alexandra Hartman, and Robert Blair, Working paper, October 2012. Annette N. Brown www.3ieimpact.org
The intervention http://reedsinthewind.blogspot.com/2012/10/week-four-not-cote-divoire-but- liberia.html Annette N. Brown www.3ieimpact.org
The counterfactual Table 2: Impacts on land disputes (adapted) All residents Residents with a dispute Any Results Any un- Results Resolved Resolved Satisfied serious in resolved in informal with dispute violence dispute violence mech- outcome anism Treatment 0.005 -0.010 -0.020** -0.028 0.080*** 0.042* 0.065* Mean, control 0.221 0.122 0.0698 0.554 0.683 0.193 0.578 group ATE as % 2% -8% -28% -5% 12% 22% 11% of control # obs 5,411 5,411 5,411 1,210 1,210 1,210 1,210 R-squared 0.173 0.145 0.063 0.121 0.073 0.065 0.085 Annette N. Brown www.3ieimpact.org
The factual http://globalvoicesonline.org/2011/11/20/liberia-giving-free-press-a-second- chance/ Annette N. Brown www.3ieimpact.org
The factual “Those who participated in the workshop, now they are available and it is okay for a person to go to them, and these people use the same skills they learned in the workshop and talk to both people involved in the dispute to solve it.” (p. 33) Annette N. Brown www.3ieimpact.org
What we don’t know • Whether the intervention changes norms • Whether the intervention changes attitudes • Longer term effects, both the progression of resolutions over time and whether the impacts of the trainings are sustainable long after the intervention is completed Annette N. Brown www.3ieimpact.org
What we think “The intervention in Liberia did not reduce the level or severity of conflict in the space of one or eighteen months, but it did change dramatically the manner and success of conflict resolution, especially with the most common and important form of conflict: land disputes.” (p. 36) Annette N. Brown www.3ieimpact.org
Evidence II: Roads “Better roads lower transaction costs associated with agricultural activities and in doing so have the potential to reduce the costs of acquiring inputs, increase output prices, reduce the impact of shocks, and permit entry into new, more profitable activities.” -Stefan Dercon, Daniel O. Gilligan, John Hoddinott, and Tassew Woldehanna , “The Impact of Agricultural Extension and Roads on Poverty and Consumption Growth in Fifteen Ethiopian Villages,” American Journal of Agricultural Economics 91(4) (November 2009) -Stefan Dercon and John Hoddinott , “Livelihoods, Growth, and Links to Market Towns in 15 Ethiopian Villages,” FCND Discussion Paper 194, July 2005. Annette N. Brown www.3ieimpact.org
The intervention http://web.worldbank.org/WBSITE/EXTERNAL/NEWS/0,,contentMDK:21275 250~menuPK:34471~pagePK:34370~piPK:34424~theSitePK:4607,00.html Annette N. Brown www.3ieimpact.org
The counterfactual Table 3. Determinants of Consumption Growth and Poverty Status: Basic Results (adapted) Consumption Poor Growth Log consumption -0.131** -0.396** Access to all- -0.069** 0.163** weather road Sample size 4,771 4,771 Annette N. Brown www.3ieimpact.org
The factual http://www.lilsoak.com/?tag=fair-trade-jewelry Annette N. Brown www.3ieimpact.org
What we don’t know The evaluation does not explore the non- economic linkages between villages and local market towns (e.g. access to healthcare, government officials, etc.) that may account for some of the decrease in poverty and/or increase in consumption growth. Annette N. Brown www.3ieimpact.org
What we think “Better roads in these localities make it easier for households to access local market towns, which in turn are linked to larger urban centers.” (p. 1018) “Public investments have the potential to play important roles in facilitating increased growth and faster poverty reduction.” (p. 1019) Annette N. Brown www.3ieimpact.org
Evidence III: Sustainability “The difficulty of scaling up is particularly pressing when the programme is complex, where the intervention stretches over time, and where the organisational setting and skills of the service providers matter greatly for the quality of the services provided.” (p. 809) Lars Ivar Oppedal Berge, Kjetil Bjorvatn, Kartika Sari Juniwaty, and Bertil Tungodden , “Business Training in Tanzania: From Research- driven Experiment to Local Implementation,” Journal of African Economies , Vol. 21, number 5 (2012). Annette N. Brown www.3ieimpact.org
The intervention Promotion of Rural Initiatives and Development Enterprises (PRIDE) http://info.textileexchange.org/te-farm-blog/bid/119393/5-Ambitions-for- Organic-Cotton-in-2012 Annette N. Brown www.3ieimpact.org
The counterfactual Figure 3. Business Knowledge Annette N. Brown www.3ieimpact.org
The factual Figure 1. Attendance per Session. Annette N. Brown www.3ieimpact.org
The factual Figure 2. Subjective Evaluation Annette N. Brown www.3ieimpact.org
What we don’t know • We do not have measures of the final business impacts — the incomes of the entrepreneurs or the revenues and profits from their businesses. • We do not know why the internal training performed poorly relative to the external training. Characteristics of trainers? Perceptions of quality going in? Annette N. Brown www.3ieimpact.org
What we think “We conclude that the estimated effect of research-led interventions should be interpreted as an upper bound of what can be achieved when scaling up such interventions locally.” (p. 809) Annette N. Brown www.3ieimpact.org
Conclusion Good impact evaluation studies can help us to understand what works, how, and why by examining both counterfactual and factual evidence. Annette N. Brown www.3ieimpact.org
Recommend
More recommend
