Studying Variation in the Effect of Program Participation Stephen - PowerPoint PPT Presentation

Studying Variation in the Effect of Program Participation Stephen W. Raudenbush Presentation at the “Workshop on Learning from Variation in Program Effects” Palo Alto, July 19, 2016 The research reported here was supported by a grant from the W.T. Grant Foundation to the University of Chicago entitled “ Building Capacity for Evaluating Group-Level Interventions. ” Thanks to Sean Reardon and Takako Nomi for their collaboration on these ideas.

Outline 1. Pervasiveness of • Multi-site trials • Non-compliance 2. Instrumental variables in a single-site study • Under homogeneity of impact • Under heterogeneity of impact • Examples 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized least squares • Method 2= “Between-Site Regression!” 4. Design Considerations 5. Modeling Program Participation and Program Impact on Participants

1. Pervasiveness of Multi-Site Trials Since 2002, IES has funded 175 group-randomized trials Vast majority are multi-site trials (Spybrook, 2013) Other recent examples * National Head Start Evaluation (US Dept of HHS, 2010) * Moving to Opportunity (Sonbanmatsu, Kling, Duncan, Brooks Gunn, 2006) * School-based lottery studies ( Abdulkadiroglu, Angrist, Dynarski, Kane, and Pathak, 2009 ). * Tennessee STAR (Finn and Achilles, 1990) * Ending Social Promotion (Jacob and Lefgren, 2009) * Double-Dose Algebra (Nomi and Allensworth, 2009) * Welfare to Work (Bloom, Hill, Riccio, 2003) * Small Schools of Choice (MDRC)

Some Recent MS Trials Study Levels Assigned Sites Fixed or Random Units sites National Head 2 Children 198 Program Random Start Eval. Sites Moving to 2 Families 5 cities Fixed Opportunity Boston 2 Children Lottery pools Random Charter School Lotteries Tennessee 3 Teachers 79 Schools Random STAR 4 R’s 3 Classrooms 18 Schools Random Double-Dose 2 Children 60 Schools Random Algebra

2. Estimating the Impact of Program Participation in a One- Site Study

Figure 1: Conventional Instrumental Variable Model (Homogeneous Treatment Effects)  T=Random M=Participation assignment    Total ITT Effect : Y=outcome    so         / 0

Single site, heterogeneous treatment effects Person-specific Causal Model  )   ( E T M  )   ( E Y    B i i i        ( ) ( , ) E B Cov

“No Compliance-Covariance” Assumption is Strong!        ( ) ( , ) E B Cov      ( , ) 0 if Cov      / 0 if

Alternative Approach for binary M “Local Average Treatment Effect” (LATE) or “Complier Average Treatment Effect” (Bloom, 1984; Angrist, Imbens, and Rubin, 1996)

Principal Stratification Stratum M(1) M(0) Г=M(1)-M(0) Y(M(1))-Y(M(0)) Fraction Average of pop Effect Compliers 1 0 1 Y(1)=Y(0) γ compliers δ compliers Always- 1 1 0 Y(1)-Y(1)=0 γ always 0 takers Never- 0 0 0 Y(0)-Y(0)=0 γ never 0 takers Defiers 0 1 -1 Y(0)-Y(1) 0 0

Complier-average treatment effect (“Local average treatment effect”)          ( ) 0 * 0 * E B compliers compliers always never      compliers compliers so     /       ( | 1 ) ( | 0 ) " " Note E Y T E Y T ITT on Y       ( | 1 ) ( | 0 ) " " E M T E M T ITT on M if T is randomized

In Sum We can estimate the Population-Average Effect of Participating if we assume Cov( Г, Δ)=0 We can estimate LATE if we assume Pr( Г≥0)=1 The latter is a weaker assumption, but does not eliminate the selection problem!

Multiple Sites How do we take this to multiple sites to * Estimate average Impact of Program Participation * Estimate variation in the Impact of Program Participation * Two methods using simulated data: “Small Schools of Choice” Design (J=200, 80<n<120) 2     ( : 1 . 194 , 0 . 430 ) true values 

Method 1: Combine 2 ITT analyses Step 1: Estimate the Impact of Treatment Assignment on the Outcome      . ( . ) . Y Y B T T e e ij j j ij j ij j     2 ( ) , ( ) E B Var B B Results ˆ    0 . 770 , 0 . 53 se      2 2 ˆ 0 . 249 . 499 B 95 % plausible value interval for B j     . 770 1 . 96 * . 499 ( . 21 , 1 . 75 )

Step 2: Estimate the Impact of Treatment Assignment on Program Participation        . ( . ) . M M G T T ij j j ij j ij j     2 ( ) , ( ) E G Var G G Results   ˆ 0 . 703      2 2 ˆ 0 . 0061 . 078 G 95 % plausible value interval for G j  (. 55 , 86 )

Step 3: Combine Results     / ˆ ˆ       ˆ / . 770 / . 703 1 . 095 In our case     2 2 2        2 if G D    2 2 D  In our case  2 2 (. 249 ) ( 1 . 095 ) * (. 078 )    ˆ 2 . 483  2 2 D (. 078 ) (. 703 )     95 % ( ) 1 . 095 1 . 96 * . 483 ( . 267 , 1 . 248 ) PV D

In sum True Values 2     : 1 . 19 , 0 . 430 True values  Our estimates 2  ˆ    ˆ 1 . 10 , 0 . 487 

Method 2: Two-Stage Generalized Least Squares: Theoretical Model        Stage 1 : . ( . ) . M M G T T ij j j ij j ij j     2 ( ) , ( ) E G Var G G        Stage 2 : . ( . ) . Y Y D M M ij j j ij j ij j     2 ( ) , ( ) E D Var D D

Method 2 in Practice ˆ ˆ    1 . : . ( . ) Compute M M G T T ij j OLS ij j ˆ        * * 2 . : . ( . ) Do HLM Y Y D M M ij j j ij j ij ij     2 ( ) , ( ) E D Var D D 3 . Recompute variances using original M ˆ        * * . ( . ) Y Y D M M ij j j ij j ij ij ˆ     2 ( ) , ( ) E D Var D D

Method 2=“Between Site Regression!”  B D G j j j      ( ) G D G j j j ..... from which ˆ ˆ ˆ          ( ) E C B G D G j j j j j j ˆ    [ ( )] 0 Requires E G j D j

Results ˆ    1 . 095 , 0 . 0730 se   2 ˆ . 4856   2 . 6968  95 % (. 316 , . 746 ) CI  2 2 (. 562 , . 863 )

Envisioning Variation: LATE Effect “Head Start” Design (J=200, 10<n<20) “Small Schools of Choice” Design (J=200, 80<n<120) “Welfare to Work” Design (J=60, 200<n<1400)

Program Participation Model (“LATE”) : Participat ion model      ( . ) . M M G T T v v ij j j ij j ij j     2 ( ) , ( ) E G Var G  : Impact Model        . ( . ) . Y Y D M M ij j j ij j ij j   2 ~ ( , ) D N  j ˆ    2  ˆ : 0 . 986 0 HS  ˆ    2  ˆ 1 . 095 , 0 . 487 SSC  ˆ    2  ˆ : 1 . 007 , 0 . 280 WtW     2  ( : 1 . 194 , 0 . 430 ) true values 

Profile Likelihood for LATE: “HS” Design 3.00 2.00 Beta 1.00 0 -1.00 0 0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01 Tau

Profile Likelihood for LATE: “SSOC” Design 2.50 Beta 1.00 -0.50 0 0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01 Tau

Profile Likelihood for LATE: “WtW” Design 3.00 2.00 Beta 1.00 0 -1.00 0 0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01 Tau

Posterior intervals for site-specific LATE Effects

Posterior Intervals for LATE: “HS” Design 2.72 1.90 MHAT 1.09 0.27 -0.55 0 50.50 101.00 151.50 202.00

Posterior Intervals for LATE: “SSC” Design 3.12 2.04 MHAT 0.96 -0.13 -1.21 0 50.50 101.00 151.50 202.00

Posterior Intervals for LATE: “W to W” Design 2.73 1.95 MHAT 1.18 0.41 -0.37 0 15.50 31.00 46.50 62.00

Moving Toward Explanation modeling participation, modeling impact B  G D j j j Total Impact of Assignment= Impact of Assignment on Participation * Impact of participation on Outcome Within site: Which persons are most likely to participate? Which persons are most likely to benefit from participation? The picture can't be displayed. Between Sites: How do we improve site-average participation rate? How do we enhance average benefit of participating? Models are needed at both levels because sites vary not only in organizational effectiveness but also in client composition