Developmental Evaluations March 2020 This presentation is made - PowerPoint PPT Presentation

Evaluation Café: A Study of Three Developmental Evaluations March 2020 This presentation is made possible by the support of the American People through the United States Agency for International Development (USAID.) The contents of this presentation are the sole responsibility of DEPA-MERL Consortium and do not necessarily reflect the views of USAID or the United States Government. 1

Who we are 2

Before we begin, • How familiar are you with Developmental Evaluation (DE)? • How familiar are you with Outcome Harvesting? • How many are interested in international evaluation work? 3

Our agenda today, • Introduction to DEPA-MERL • Overview of research questions & methods • Key lessons learned • Audience questions 4

Introduction to DEPA-MERL • WDI working with USAID since 2015 • MERLIN Program: – Innovate on traditional approaches to monitoring, evaluation, research and learning (MERL) • DEPA-MERL is a consortium under MERLIN that implements developmental evaluations in the USAID context 5

What is developmental evaluation? • Supports the continuous adaptation of development interventions in complex environments • Developmental Evaluators are “embedded” within the program – Use a variety of M&E approaches 6

How we applied DE • Three DE pilots were conducted – Family Care First in Cambodia – Sustained Uptake DE, the US Global Development Lab (Washington, DC) – Bureau for Food Security (Washington, DC) • Each DE pilot included: – 1 full-time, external Developmental Evaluator – A “DE Administrator” from DEPA -MERL 7

Our research questions Research question Methods Data sources INSERT DE RQ QUESTIONS TABLE FROM IP AAP REPORT ● Monthly reflection interviews with two Developmental 1: How does DE capture, promote, Outcome and enable the utilization of emergent harvesting Evaluators (n=35) learnings in support of ongoing (qualitative) ● Substantiation interviews with keys stakeholders at programming in a complex system, in endline of two DEs (n=26) the USAID context? ● Document reviews , as required 2: What are the barriers and enablers Semi- ● Monthly reflection interviews with two Developmental to implementation of DE in the USAID structured Evaluators (n=35) context? interviews ● Substantiation interviews with keys stakeholders at (qualitative) endline of two DEs (n=26) 3: What is the perceived value of Survey ● Value of Developmental Evaluation Survey with DE conducting a DE , especially versus a (quantitative stakeholders at endline (n=30) traditional evaluation approach? and qualitative) 8

Outcome harvesting approach 9

Ben White What was the most surprising finding from our research on DE? 10

What is the biggest difference between DE in theory and DE in practice? 11

If you were approached by new DE implementer, what advice would you give them about how to successfully launch and oversee a DE? https://bit.ly/32kJGOA 12

Diego PH What is the most controversial idea or topic in DE right now? 13

Resources! https://bit.ly/2JXCUru https://bit.ly/2VkTbgO 14

Questions? 15

Contact us! WDI Website USAID-MERLIN (DEPA-MERL) Shannon Griswold, sgriswold@usaid.gov WDI WDI-performancemeasurement@umich.edu https://bit.ly/2NtuVU7 16

Annex 17

What is developmental evaluation? • Development Evaluation (DE) is an approach to evaluation that supports the continuous adaptation of development interventions . • DE provides evaluative thinking and timely feedback to inform ongoing adaptation as needs, findings, and insights emerge in complex dynamic situations. • The DE helps facilitate the process from findings to action in a collaborative process with the DE stakeholders. 18

What outcomes can you expect from DE? Realized Outcomes Unrealized Outcomes Intended Implemented Outcomes Outcomes Emergent Outcomes Source: Henry Mintzberg, Sumatra Ghoshal, and James B. Quinn, The Strategy Process , Prentice Hall, 1998.

How is DE different? Traditional Evaluation Developmental Evaluation Purpose Supports improvement, Supports development of innovation and summative test and adaptation in dynamic environments accountability Standards Methodological competence Methodological flexibility and and commitment to rigor, adaptability; systems thinking, creative independence, credibility with and critical thinking balanced; high external authorities tolerance for ambiguity; able to facilitate rigorous evidence-based perspectives Options Traditional research and Utilization focused: options are chosen in disciplinary standards of quality service to developmental use dominate options Evaluation Detailed formal reports; Rapid, real time feedback; diverse, user- Results validated best practices, friendly forms of feedback. Evaluation generalizable across time and aims to nurture learning. space. 20

What is a developmental evaluator? • The DE evaluator works collaboratively with implementing teams to conceptualize, design, and test new approaches in a long-term, ongoing process of adaptation, intentional change, and development. • The DE evaluator thinks and engages evaluatively ; questions assumptions; applies evaluation logic; uses appropriate methods; and stays empirically grounded — that is, rigorously gathers, interprets, and reports data. 21

Analysis Categories for the Harvested Outcomes (RQ1) 22

Barriers and Enablers (RQ2) Factor Percent of all enablers* Percent of all barriers* Skills of the Developmental Evaluator 17% 8% Data collection and sharing 16% 10% Data utilization 12% 8% Integration of the Developmental Evaluator 11% 10% Leadership (of program being evaluated) 11% 15% Stakeholder relationships 9% 11% DE readiness 8% 8% USAID dynamics 7% 14% Funding dynamics 2% 6% Local and international dynamics 2% 4% * Percentages do not total 100% because only 10 of the 13 factors that were coded across DE pilots for are shown. 23

Value of Developmental Evaluation Survey (RQ3) (1/2) Takeaway: In all areas except DE cost and time savings, survey respondents said the DEs were better than traditional evaluation (n=29) 24

Value of Developmental Evaluation Survey (RQ3) (2/2) Takeaway: The overwhelming majority of respondents reported positive interactions with their Developmental Evaluators. 25

Description of codes (1/2) 26

Description of codes (2/2) 27

The DEPA-MERL DE pilots 28