Doing More When Youre Running LATE: Applying Marginal Treatment - PowerPoint PPT Presentation

Doing More When You’re Running LATE: Applying Marginal Treatment Effect Methods to Examine Treatment Effect Heterogeneity in Experiments Amanda E. Kowalski Associate Professor, Department of Economics, Yale Faculty Research Fellow, NBER October 2016

“Doing More When You’re Running LATE: Applying Marginal Treatment Effect Methods to Examine Treatment Effect Heterogeneity in Experiments.” NBER WP 22363. “How to Examine External Validity Within an Experiment” Prepared for JEP. 2 of 67

Are experimental results externally valid? • I examine treatment effect heterogeneity within an experiment 3 of 67

Are experimental results externally valid? • Build on selection/moral hazard in insurance – Einav, Finkelstein, and Cullen (2010) – Hackmann, Kolstad, and Kowalski (2015) • Build on MTE and LATE – Bjorklund and Moffitt (1987) – Imbens and Angrist (1994) – Heckman and Vytlacil (1999, 2005, 2007) – Carneiro, Heckman, and Vytlacil (2011) – Brinch, Mogstad, Wiswall (2015) 4 of 67

I examine selection and treatment effect heterogeneity in experiments 1. Heterogeneity across unobservables – Test for selection – Bound – Estimate – Generalize comparison of OLS to IV 2. Heterogeneity across observables – Characterize – Extrapolate Application: Oregon Health Insurance Experiment 5 of 67

6 of 67

7 of 67

8 of 67

9 of 67

0.00 1.00 U D : net unobserved cost of treatment 10 of 67

𝑎 = 0 D= D=1 0 ≤ 𝑉 $ ≤ 𝑞 & Al Alway ays Tak akers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 11 of 67

𝑎 = 0 D=1 D= D=0 0 ≤ 𝑉 $ ≤ 𝑞 & p & < 𝑉 $ ≤ 1 Always Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 12 of 67

𝑎 = 1 D= D=0 p , < 𝑉 $ ≤ 1 𝑎 = 0 D=1 D=0 0 ≤ 𝑉 $ ≤ 𝑞 & p & < 𝑉 $ ≤ 1 Always Never Ne Takers Tak akers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 13 of 67

𝑎 = 1 D= D=1 D=0 0 ≤ 𝑉 $ ≤ 𝑞 , p , < 𝑉 $ ≤ 1 𝑎 = 0 D=1 D=0 0 ≤ 𝑉 $ ≤ 𝑞 & p & < 𝑉 $ ≤ 1 Always Never Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 14 of 67

𝑎 = 1 D=1 D=0 0 ≤ 𝑉 $ ≤ 𝑞 , p , < 𝑉 $ ≤ 1 𝑎 = 0 D=1 D=0 0 ≤ 𝑉 $ ≤ 𝑞 & p & < 𝑉 $ ≤ 1 Al Alway ays Com ompliers Ne Never Tak akers Tak akers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 15 of 67

100 Treated Outcome Untreated Outcome 56 28 0 Always Compliers Never Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 𝒂 = 𝟏 D= D=1 D=0 D= 16 of 67

100 Treated Outcome Untreated Outcome 61 56 28 23 0 Always Compliers Never Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 𝑎 = 0 D=1 D=0 𝒂 = 𝟐 D=1 D= D=0 D= 17 of 67

100 Treated Outcome Untreated Outcome 68 61 56 36 28 23 0 Always Never Com ompliers Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 𝑎 = 0 D=1 D=0 𝑎 = 1 D=1 D=0 18 of 67

100 Treated Outcome Untreated Outcome (0.60 ∗ 61 − 0.35 ∗ 56) 0.60 − 0.35 68 61 56 ((1 − 0.35) ∗ 28 − (1 − 0.60) ∗ 23) 0.60 − 0.35 36 28 23 0 Always Never Com ompliers Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 𝑎 = 0 D=1 D=0 𝑎 = 1 D=1 D=0 19 of 67

100 Treated Outcome Untreated Outcome 68 36 0 Always Never Com ompliers Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 20 of 67

100 Treated Outcome Untreated Outcome Treatment Effect 68 LATE 36 0 Always Never Com ompliers Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 21 of 67

100 Treated Outcome Untreated Outcome, Selection Effect Treatment Effect 68 56 LATE 36 23 0 Compliers Al Alway ays Never Ne Tak akers Tak akers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 22 of 67

I examine selection and treatment effect heterogeneity in experiments 1. Heterogeneity across unobservables – Test for selection – Bound – Estimate – Generalize comparison of OLS to IV 2. Heterogeneity across observables – Characterize – Extrapolate Application: Oregon Health Insurance Experiment 23 of 67

24 of 67

25 of 67

26 of 67

100 Treated Outcome Untreated Outcome, Selection Effect Treatment Effect 68 56 LATE 36 lower bound 23 0 Compliers Never Al Alway ays Takers Tak akers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 27 of 67

100 Treated Outcome Untreated Outcome, Selection Effect Treatment Effect 68 56 LATE upper bound 36 lower bound 23 0 Compliers Never Al Alway ays Takers Tak akers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 28 of 67

100 Treated Outcome Untreated Outcome, Selection Effect Treatment Effect lower bound 68 56 LATE upper bound lower bound 36 lower bound 23 0 Always Compliers Never Ne Takers Tak akers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 29 of 67

100 Treated Outcome Untreated Outcome, Selection Effect 86 Treatment Effect upper bound 68 upper bound LATE upper bound 36 lower bound 23 0 Always Compliers Never Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 30 of 67

I examine selection and treatment effect heterogeneity in experiments 1. Heterogeneity across unobservables – Test for selection – Bound – Estimate – Generalize comparison of OLS to IV 2. Heterogeneity across observables – Characterize – Extrapolate Application: Oregon Health Insurance Experiment 31 of 67

100 Treated Outcome Untreated Outcome, Selection Effect 86 Treatment Effect upper bound 68 LATE 36 lower bound 23 0 Always Compliers Never Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 32 of 67

100 MTO(p), Marginal Treated Outcome MUO(p), Marginal Untreated Outcome, Marginal Selection Effect 86 MTE(p), Marginal Treatment Effect upper bound 68 LATE 36 lower bound 23 0 Always Compliers Never Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment p: fraction treated 33 of 67

100 MTO(p), Marginal Treated Outcome MUO(p), Marginal Untreated Outcome, Marginal Selection Effect 86 MTE(p), Marginal Treatment Effect 68 LATE 36 23 0 Always Compliers Never Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment p: fraction treated 34 of 67

100 MTO(p), Marginal Treated Outcome MUO(p), Marginal Untreated Outcome, Marginal Selection Effect 86 MTE(p), Marginal Treatment Effect 68 36 23 0 Always Compliers Never Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment p: fraction treated 35 of 67

I examine selection and treatment effect heterogeneity in experiments 1. Heterogeneity across unobservables – Test for selection – Bound – Estimate – Generalize comparison of OLS to IV 2. Heterogeneity across observables – Characterize – Extrapolate Application: Oregon Health Insurance Experiment 36 of 67

100 86 68 Baseline OLS 36 28 23 0 Always Compliers Never Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 𝑎 = 0 D=1 D=0 37 of 67

100 86 78.5 68 Intervention OLS 36 28 23 0 Always Compliers Never Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 𝑎 = 1 D=1 D=0 38 of 67

100 86 78.5 68 Baseline OLS LATE Intervention OLS 36 28 23 0 Always Compliers Never Takers Takers 0.00 1.00 p B =0.35 p I =0.60 U D : net unobserved cost of treatment 𝑎 = 0 D=1 D=0 𝑎 = 1 D=1 D=0 39 of 67

I examine selection and treatment effect heterogeneity in experiments 1. Heterogeneity across unobservables – Test for selection – Bound – Estimate – Generalize comparison of OLS to IV 2. Heterogeneity across observables – Characterize – Extrapolate Application: Oregon Health Insurance Experiment 40 of 67

Quantify unexplained variation 1 0 (MTE p − ATE)2dp = 5.77 RMSD= 0 100 MTE(p), Marginal Treatment Effect ATE 31.5 0 Always Compliers Never Takers Takers 0.00 p B =0.35 p I =0.60 1.00 U D : net unobserved cost of treatment p: fraction treated 41 of 67

42 of 67

43 of 67

44 of 67

45 of 67

Explain variation with observables 100 MTE(p), Marginal Treatment Effect SMTE(p), with covariates minMTE(p), with covariates maxMTE(p), with covariates 0 Always Compliers Never Takers Takers 0.00 p B =0.35 p I =0.60 1.00 U D : net unobserved cost of treatment p: fraction treated 46 of 67