Ra ndo mize d Co ntro lle d T ria ls, De ve lo pme nt E c o no mic s a nd Po lic y Ma king in De ve lo ping Co untrie s Esthe r Duflo De p a rtm e nt o f Ec o no m ic s, MIT C o -Dire c to r J-PAL [Jo int w o rk w ith Ab hijit Ba ne rje e a nd Mic ha e l Kre m e r]
Ra ndo mize d c o ntro lle d tria ls ha ve g re a tly e xpa nde d in the la st two de c a de s Ra ndo mize d c o ntro lle d T ria ls we re pro g re ssive ly • a c c e pte d a s a to o l fo r po lic y e va lua tio n in the US thro ug h ma ny b a ttle s fro m the 1970s to the 1990s. I n de ve lo pme nt, the ra pid g ro wth sta rts a fte r the mid • 1990s – K re me r e t a l, studie s o n K e nya (1994) – PROGRE SA e xpe rime nt (1997) Sinc e 2000, the g ro wth ha ve b e e n ve ry ra pid. • J - PAL | T 2 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
Ca me ro n e t a l (2016): RCT in de ve lo pme nt Figure 1: Number of Published RCTs 300 250 200 150 100 50 0 1975 1980 1985 1990 1995 2000 2005 2010 2015 Pub lic a tio n Ye a r J - PAL | T 3 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
BRE AD Affilia te s do ing RCT Figure 4. Fraction of BREAD Affiliates & Fellows with 1 or more RCTs 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1980 or earlier 1981-1990 1991-2000 2001-2005 2006-today PhD Year * Total Number of Fellows and Affiliates is 166 . J - PAL | T 4 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
T o p Jo urna ls J - PAL | T 5 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
Ma ny se c to rs, ma ny c o untrie s J - PAL | T 6 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
Why ha ve RCT ha d so muc h impa c t? F o c us o n ide ntific a tio n o f c a usa l e ffe c ts (a c ro ss the • b o a rd) Asse ssing E xte rna l Va lidity • Ob se rving Uno b se rva b le s • Da ta c o lle c tio n • I te ra tive E xpe rime nta tio n • Unpa c k impa c ts • J - PAL | T 7 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
F o c us o n I de ntific a tio n… a c ro ss the b o a rd! T he ke y a dva nta g e o f RCT wa s pe rc e ive d to b e a c le a r • ide ntific a tio n a dva nta g e With RCT , sinc e tho se who re c e ive d a tre a tme nt a re • ra ndo mly se le c te d in a re le va nt sa mple , a ny diffe re nc e b e twe e n tre a tme nt a nd c o ntro l must b e due to the tre a tme nt Mo st c ritic isms o f e xpe rime nt a lso fo c us o n limits to • ide ntific a tio n (impe rfe c t ra ndo miza tio n, a ttritio n, e tc . ) o r thing s tha t a re no t ide ntifie d e ve n b y ra ndo mize d tria ls (distrib utio n o f tre a tme nt e ffe c ts, e ffe c ts e lse whe re ). J - PAL | T 8 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
F o c us o n I de ntific a tio n… a c ro ss the b o a rd! Be fo re the e xplo sio n o f RCT in de ve lo pme nt, a lite ra ture • o n RCT in la b o r a nd pub lic fina nc e ha s tho ug ht o f o the r wa ys to ide ntify c a usa l e ffe c ts I n de ve lo pme nt e c o no mic s, the re wa s a jo int • de ve lo pme nt o f the two lite ra ture s (na tura l e xpe rime nt a nd RCT ), whic h ha s ma de b o th lite ra ture s stro ng e r, a nd pe rha ps le ss diffe re nt tha n we initia lly tho ug ht the y wo uld b e : – Na tura l e xpe rime nts think o f RCT a s a na tura l b e nc hma rk (no t just a n hypo the tic a l g o ld sta nda rd). – De ve lo pme nt o f me tho ds to g o b e yo nd simple c o mpa riso n o f tre a tme nt a nd c o ntro l in e xpe rime nts, a nd ric he r de sig ns J - PAL | T 9 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
E nc o ura g e me nt de sig n Pe o ple who ta ke up pro g ra m Diffe re nc e in ta ke up c a use d b y e nc o ura g e me nt
F o c us o n I de ntific a tio n… a c ro ss the b o a rd! Be fo re the e xplo sio n o f RCT in de ve lo pme nt, a lite ra ture o n • RCT in la b o r a nd pub lic fina nc e ha s tho ug ht o f o the r wa ys to ide ntify c a usa l e ffe c ts I n de ve lo pme nt e c o no mic s, the re wa s a jo int de ve lo pme nt o f • the two lite ra ture s (na tura l e xpe rime nt a nd RCT ), whic h ha s ma de b o th lite ra ture s stro ng e r, a nd pe rha ps le ss diffe re nt tha n we initia lly tho ug ht the y wo uld b e : Na tura l e xpe rime nts think o f RCT a s a na tura l b e nc hma rk (no t just a n – hypo the tic a l g o ld sta nda rd). E xtre me ly we ll ide ntifie d no n ra ndo mize d studie s. De ve lo pme nt o f me tho ds to g o b e yo nd simple c o mpa riso n o f – tre a tme nt a nd c o ntro l in e xpe rime nts, a nd ric he r de sig ns Ultima te ly, the a dva nta g e o f RCT in te rms o f ide ntific a tio n is a • ma tte r o f de g re e , ra the r tha n a funda me nta l diffe re nc e . J - PAL | T 11 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
Why ha ve RCT ha d so muc h impa c t? F o c us o n ide ntific a tio n o f c a usa l e ffe c ts (a c ro ss the • b o a rd) Asse ssing E xte rna l Va lidity • Ob se rving Uno b se rva b le s • Da ta c o lle c tio n • I te ra tive E xpe rime nta tio n • Unpa c k impa c ts • J - PAL | T 12 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
E xte rna l Va lidity Will re sults o b ta ine d so me whe re g e ne ra lize e lse whe re ? • A fre q ue nt c ritic ism o f RCT is tha t the y do n’ t g ua ra nte e • e xte rna l va lidity Whic h is q uite rig ht, b ut it is no t like the y a re le ss • e xte rna lly va lid… And b e c a use the y a re inte rna lly va lid, a nd b e c a use yo u • c a n c o ntro l whe re the y will ta ke pla c e : – c o mpa re d a c ro ss c o nte xts. the y c a n b e purpo se fully run in diffe re nt c o nte xts – – Pre dic tio n c a n b e ma de o f wha t the e ffe c ts o f re la te d pro g ra ms c o uld b e . J - PAL | T 13 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
Ba ye sia n Hie ra rc hic a l Mo de lling o f a ll the MF re sults : Pro fits Me a g e r (2015)
Ba ye sia n Hie ra rc hic a l Mo de ling -- Me ta a na lysis (c o nsumptio n) 15
E xa mple 2: T a rg e ting the Ultra Po o r Pro g ra m: Co o rdina te d e va lua tio n in se ve ra l c o untrie s Productive asset transfer Savings Health Beneficiary Home visits Consumption support Technical skills training Ba ne rje e e t a l, 2015 16
Co untry b y c o untry re sults: Asse ts Endline 1 Endline 2 0.8 Asse t c ha ng e (sta nda rd de via tio ns) 0.5 0.2 -0.1 Ba ne rje e e t a l, 2015 17
Co untry b y c o untry re sults: Co nsumptio n 20% Endline 1 Endline 2 15% % Cha ng e in pe r c a pita c o nsumptio n 10% 5% 0% -5% 18
Struc ture d Spe c ula tio n Ultima te ly, if the re sults a re simila r it is nic e , b ut if the y a re • diffe re nt the e x-po st a na lysis is spe c ula tive . Ba ne rje e , Cha ssa ng , Sno wb e rg (2016) pro po se to b e • e xplic it a b o ut suc h spe c ula tio n, a nd tha t re se a rc he rs sho uld pre dic t wha t the e ffe c t ma y b e fo r o the r inte rve ntio ns, o r in o the r c o nte xts. T his c a n the n mo tiva te running suc h e xpe rime nts, a nd • g ue sse s c a n b e fa lsifie d. E xa mple : Dupa s (2014)—E ffe c t o f sho rt run sub sidie s o n • lo ng run a do ptio n de pe nd o n the timing o f c o sts a nd b e ne fits, a nd ho w q uic kly unc e rta inty a b o ut the m is re so lve d: this a llo ws he r to c la ssify the g o o ds. J - PAL | T 19 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
Why ha ve RCT ha d so muc h impa c t? F o c us o n ide ntific a tio n o f c a usa l e ffe c ts (a c ro ss the • b o a rd) Asse ssing E xte rna l Va lidity • Ob se rving Uno b se rva b le s • Da ta c o lle c tio n • I te ra tive E xpe rime nta tio n • Unpa c k impa c ts • J - PAL | T 20 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
Ob se rving uno b se rva b le s So me thing s simply c a nno t b e o b se rve d in the wild, with • na tura lly o c c urring va ria tio n Ne g a tive inc o me ta x e xpe rime nt wa s de sig ne d a s a n • e xpe rime nt to se pa ra te inc o me a nd sub stitutio n e ffe c ts Ma ny e xpe rime nts in de ve lo pme nt a re de sig ne d like wise • to c a pture suc h e ffe c ts: – K a rla n Zinma n Ob se rving Uno b se rva b le s – Co he n Dupa s a nd Ashra f Dupa s Sha piro : se le c tio n a nd tre a tme nt e ffe c t o f pric e s. – Be rtra nd e t a l. Co rruptio n in driving lic e nc e s in De lhi. J - PAL | T 21 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
Why ha ve RCT ha d so muc h impa c t? F o c us o n ide ntific a tio n o f c a usa l e ffe c ts (a c ro ss the • b o a rd) Asse ssing E xte rna l Va lidity • Ob se rving Uno b se rva b le s • Da ta c o lle c tio n • I te ra tive E xpe rime nta tio n • Unpa c k impa c ts • J - PAL | T 22 HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y
Recommend
More recommend
