the Impact of Microfinance and Informal Sector Borrowing in - PowerPoint PPT Presentation

Replication, Reproduction and the Credibility of Micro-econometric Studies of the Impact of Microfinance and Informal Sector Borrowing in Bangladesh Maren Duvendack 1, 2 Richard Palmer-Jones 1 6 June 2012 1. UEA, 2. ODI 1

Replication & impact heterogeneity  Many good reasons to conduct replications, one is to explore impact heterogeneity:  Crucial for drawing appropriate policy conclusions, confounds causal effects because results only hold for the groups identified, and not others  Neglecting impact heterogeneity misleading by inappropriately merging groups which respond quite differently to the treatment and indeed experience different treatments  Sub-groups known feature of MF context  Replication should seek to know the conditions under which the results hold, often by repeating the experiments  Thus, heterogeneity and replication are closely linked. We want to know what works, for whom, under what circumstances. 2

Introduction: Microfinance evaluations  Microfinance hype: MF has long been seen as silver bullet for alleviating poverty and empowering women through providing financial services to the poor  Studies suggesting social and economic benefits: Hulme and Mosley (1996), Coleman (1999), Pitt and Khandker (1998),  Khandker (1998 and 2005), Rutherford (2001) and Morduch and Haley (2002)  Critical voices: Roodman and Morduch (2011), Duvendack et al (2011), Stewart et al  (2011), Bateman (2010) and Dichter and Harper (2007), Roy (2010) First two RCTs in the sector (Banerjee et al, 2009; Karlan and Zinman,  2009) raising doubts about the causal link between MF and poverty alleviation.  Most influential MF IE to date: Pitt and Khandker (1998) 3

Why is Pitt and Khandker so important?  Methodologically innovative  Large original World Bank survey in 1991-2 With follow up panel in 1998-9   Complex and sophisticated analysis (WESML-LIML)  Most rigorous impact evaluation of microfinance  Key work of main academic author(s)  Widely cited by high level MF advocates such as M. Yunus 4

Introduction to Pitt and Khandker (1998) Iconic study finding positive impacts of MF especially when lending to  women (male and female groups) Quasi-experimental design & eligibility condition used to identify impact  Primary eligibility criterion: landownership (0.5 acres = 50 decimals)  Overall sample:1,798; 1,538 households from treatment villages, 260 from controls  “Treatment” Village “Control” Village Land owned/cultivated Would not Not-eligible non be eligible participants 0.5 acres Eligible participants Would be eligible Eligible but non participants 5

Sub-groups & Microfinance  PnK simply ignore alternative sources of finance but they appear in their data: Status Treatment villages, Control villages, no. of individuals with no. of individuals with multiple sources (in %) multiple sources (in %) Eligible 4 7 Not eligible 2.5 8  Sub-groups should have been in the design to explore impact heterogeneity  One should test against next best alternative, research design neglected alternatives  Sub-groups little analysed and not vs MF 6

Sub-group comparisons Lack of sub-group heterogeneity undermines claim that MFIs  make unique contribution to poverty reduction. 1: MF None Other MF None 2: Borr Other MF 3: Borr 7

Sub-group PSM results Y MF + Y MF vs Y MF vs Y Multiple +Y Borr Outcome variables Y None Y Borr vs Y None Comparison 1 2 3 Kernel matching, 0.05 Log per capita expenditure (Taka) -0.004 0.035*** -0.114*** Log women non-landed assets 0.846*** 0.501*** 1.183*** (Taka) Girl school enrolment, aged 5-17 0.070*** 0.073*** 0.029 years Boy school enrolment, aged 5-17 0.037 0.057*** 0.007 years • Summary: Sub- group analysis undermines PnK’s claims, no obvious advantage of MF vs other sources

Conclusions  Too easy to believe MF is beneficial without considering evidence in balanced way, thus sub-group analysis crucial  undermines PnK’s original claims, supported by other data sources  Award prestige only if public deposit of original data and code allowing replication and reproduction "The more freely researchers circulate their data and code, the easier it is for  others to subject that work to the scrutiny needed for science to proceed. The stakes are particularly high for research that influences policy“ (Roodman and Morduch, 2011:45).  Had data and code disclosure policy been in place at the time PnK was published, we might have resolved the current debate over this study a while ago (Roodman and Morduch, 2011).  Continuing need for high quality studies with ethical reporting and publication practices (enabling replication)  Ensure opportunities for independent research by people from different methodological and epistemological backgrounds. 9

Q & A Session For further questions or comments please email: m.duvendack@odi.org.uk; or r.palmer-jones@uea.ac.uk 10

Appendix: PSM Results – By Gender Y MF vs Y MF + Multiple +Y Borr Y MF vs Y Borr Outcome variables Y None vs Y None Comparison 1 2 3 Kernel matching, 0.05 Women -0.002 0.002 -0.111*** Log per capita 1 expenditure (Taka) 2 Men -0.002 0.067*** -0.121*** Women 0.948*** 0.961*** 1.510*** Log women 3 non-landed assets 4 Men 0.441 0.041 0.834** Women 0.074** 0.081*** 0.046 Girl enrolment, 5 aged 5-17 years 6 Men 0.052 0.056** 0.029 Women 0.049* 0.053** 0.040 Boy enrolment, 7 aged 5-17 years 8 Men 0.007 0.050** -0.033 Source: Authors calculations. Notes: *statistically significant at 10%, **statistically significant at 5%, ***statistically significant at 1%. Results refer to the differences in the mean values between matched samples. t-tests before and after matching employed to investigate the differences in the mean values for each covariate X across matched samples; the test provided conclusive results.

Appendix: Sensitivity Analysis  PSM estimate for log of women non-landed assets for Y MF : 0.846*** (comparison 1) - sensitive to selection on unobservables? Significance levels Hodges-Lehmann point 95% Confidence estimates intervals Gamma (Γ) Minimum Maximum Minimum Maximum Minimum Maximum 1 < 0.0001 < 0.0001 0.652 0.652 0.106 1.132 1.2 < 0.0001 < 0.4344 0.031 1.209 -0.300 1.590 1.3 < 0.0001 < 0.8037 -0.144 1.419 -0.437 1.759 1.4 < 0.0001 < 0.9641 -0.297 1.587 -0.539 1.899 Source: Authors calculations.