Capturing and Communicating Results in Complex Contributions 8 November 2012 09.15-12.00
Development Talks Susanne Wadstein Director Department for Organisational Development Sida
Development Talks Time Item 09.15 - 09.30 Welcome (Susanne Wadstein, Sida) 09.30 - 09.45 Definition of results (Lennart Peck, Sida) 09.45 - 10.30 Capturing results (Michael Woolcock, World Bank) 10.30 - 10.40 Coffee Break 10.40 - 10.55 “What about the results” (Charlotte Örnemark, Nordic Consulting Group) 10.55 - 11.10 An Example: “Vietnam, Laos & Sri Lanka: Evaluation of long - term development co- operation” (Annika Nordin Jayawardena, Sida) 11.10 - 12.00 Panel discussion & questions
RESULTS
OECD/DAC Definition of Results “The output, outcome or impact … intended or unintended… positive and/or negative… of a development intervention”
Results of what? For example: • a single project or programme • Sweden’s cooperation with a country • the joint efforts of partner country and donors • a strategy or a policy • influencing factors in the environment
Results at what point? For example: • after X number of years • “primary”, “ secondery ”, “tertiary” etc. effects • “output”, “outcome” and “impact”
Results for whom? For example: • The individual citizen • ” Women ”, ”rural population”, ”the poor ” etc. • Citizens in a village, region or country
Results in terms of what? For example: • Products, goods and services • Changes of individual or organizational behaviour, attitudes, knowledge etc. • Changes in the level of service provision, protection of human rights etc. • Changes in terms of individual wellbeing
An example: Results of electrification Results of Results at what Results in terms of Results for whom point The results of the …after completion… … were new … for 2 000 Swedish connections… households contribution … The results of joint … in an impact … was access to … for Mrs. x and her donor cooperation perspective … television, better family with government… security and improved business… The result of access … after some time… … was a reduction of … on citizens to donor funds… government’s own needing electricity spending… .
Degrees of causation • Necessary and sufficient • Necessary but not sufficient • Sufficient but not necessary • Neither sufficient nor necessary - but contributing
Different causal patterns One cause – One result One cause – Multiple results Multiple causes – One result Multiple causes – Multiple results
Attribution – Contribution – Confusion ” to contribute to create conditions that will enable poor people to improve their lives”.
Four key questions: • Did the intervention make a difference? • How has the intervention made a difference? • To what extent can a specific result be attributed to the intervention? • Will the intervention work elsewhere?
Some of our challenges • Establishing attribution/contribution • Capturing both the qualitative and the quantitative • Aggregating results • Generalising • Communicating complex things in a simple way
Development Talks Michael Woolcock Lead Social Development Specialist The World Bank
“But How Generalizable is That?” A Framework for Assessing the Internal and External Validity of Complex Development Interventions Michael Woolcock World Bank and Harvard University mwoolcock@worldbank.org SIDA, Stockholm 8 November 2012
Theory is when you know everything but nothing works. Practice is when everything works but nobody knows why. We have put together theory and practice: nothing is working… and nobody knows why! Albert Einstein
Overview • Background • From assessing IV to EV in ‘complex’ projects – Importance of trajectories, theories of change • A framework for integrating – Design elements – Causal density – Implementation dynamics – Context compatibility • Implications for EV, and for case studies
Primary source material • Bamberger, Michael, Vijayendra Rao and Michael Woolcock (2010) “Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development”, in Abbas Tashakkori and Charles Teddlie (eds.) Handbook of Mixed Methods (2 nd revised edition) Thousand Oaks, CA: Sage Publications, pp. 613-641 • Barron, Patrick, Rachael Diprose and Michael Woolcock (2011) Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia New Haven: Yale University Press • Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project Evaluation: A Contextualized Approach to Understanding Impact Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14 • Woolcock, Michael and Arathi Rao (2012) ‘But How Generalizable is That? A Framework for Assessing the External Validity of “Complex” Development Interventions’ Mimeo
Background • Rising obsession with ‘causality’, RCTs – Pushed by donors, foundations (e.g., Gates) – Yet also serious critique • In medicine: Rothwell (2005) • In philosophy: Cartwright (2011) • In economics: Deaton (2010), Heckman, Ravallion … – Especially as it pertains to EV • Engber (2011) on ‘Black 6’ (biomedical research) • Heinrich et al (2011) on ‘WEIRD’ people (social psychology) • Across time, space, groups, scale • How to assess ‘social’, ‘participatory’ projects? – Barron, Diprose and Woolcock (2011) – Mansuri and Rao (2012) – How to compare roads, irrigation, ‘empowerment’?
A typology of interventions… ‘Simple’ ‘Complicated’ ‘Complex’ ‘Chaotic’ Agriculture, Nets, pills, Education, Local justice microcredit roads health reform, CDD Theory High Predictive precision Cumulative knowledge Low Subject/object gap Mechanisms # Causal pathways Few # of ‘people - based’ Many transactions # Feedback loops Outcomes Plausible range Narrow Wide Measurement precision 22
…or aspects of interventions (e.g., health) ‘Simple’ ‘Complicated’ ‘Complex’ ‘Chaotic’ Build clinics Inoculation Ambulatory Adolescent (logistics) care sexual behavior Theory High Predictive precision Cumulative knowledge Low Subject/object gap Mechanisms # Causal pathways Few # of ‘people - based’ Many transactions # Feedback loops Outcomes Plausible range Narrow Wide Measurement precision 23
Contesting Development Participatory Projects and Local Conflict Dynamics in Indonesia PATRICK BARRON RACHAEL DIPROSE MICHAEL WOOLCOCK Yale University Press, 2011
Summary of findings Context Capacity Low High Program Functionality Program Functionality Type of Impact Low High Low High Forums (places) -- ++ --* 0 Direct Facilitators (people) 0 0 0 0 Group Relations 0 ++ + +++ Indirect Behavioral 0 +++ 0 + Normative 0 + 0 +++ * While we noted higher rates of KDP-triggered conflict in high capacity areas, such conflict is much less likely to escalate and/or turn violent. Hence negative impacts are greater in low capacity areas, where program functionality is poor.
Lessons for evaluating ‘chaotic’ projects In Evaluation 101, we assume… Impact = f (Design) | Selection, Confounding Variables Adequate for ‘simple’ interventions with a ‘good - enough’ counterfactual. But this is inadequate for assessing ‘complex’ interventions: * design is multi- faceted (i.e., has high ‘causal density’) * interaction with context is pervasive, desirable * implementation quality is vital (high discretion) * trajectories of change are probably non-linear (perhaps unknowable ex ante)
Lessons for evaluating ‘chaotic’ projects Impact = f ([DQ, CD], IE, CC) | SE, CV, RE DQ = Design quality (weak, strong) CD = Causal density (low, high), or ‘discretionary mechanisms’ (few, many; tight, loose; seen, unseen) IE = Implementation effectiveness (low, high) CC = Context compatibility (resistant, supportive) SE = Selection effects (non-random placement, participation) CV = Confounding variables RE = Reasoned expectations (where by when?) * In Social Development projects (cf. roads, immunizations): CD is high, loose, often unseen; IE and CC are variable; RE is often unknown (unknowable?)
Pervasive problem • SD projects are inherently very complex, thus: – Very hard to isolate ‘true’ impact – Very hard to make claims about likely impact elsewhere – Understanding how (not just whether) impact is achieved is also very important • Process Evaluations, or ‘Realist Evaluations’, can be most helpful (see work of Ray Pawson, Patricia Rogers et al)
From IV to EV in complex interventions: Understanding impact trajectories Net Impact t = 0 t = 1 Time
Understanding impact trajectories “Same” impact claim, but entirely Net a function of when Impact the assessment was done t = 0 t = 1 Time
Understanding impact trajectories B C A Net Impact t = 0 t = 1 Time
Understanding impact trajectories D B C A Net Impact ? t = 0 t = 1 t = 2 Time
Recommend
More recommend
