The Multi-Site RCT A fleet of RCTs! Each conducted in a different - PowerPoint PPT Presentation

The Multi-Site RCT • A fleet of RCTs! • Each conducted in a different social setting • Since 2002, IES has funded 175 randomized trials; • Vast majority are multi-site trials (Spybrook, 2013)

Opportunities • Can assess generalizability of impact – Gauge variation in effect – Model heterogeneity – Does the intervention equalize outcomes? But we must formulate appropriate analyses. This is trickier than it might seem!

Some Recent MS Trials Study Levels Assigned Sites Fixed or Random Units sites National Head 2 Children 300+ Program Random Start Eval. Sites Moving to 2 Families 5 cities Fixed Opportunity Boston 2 Children Lottery pools Random Charter School Lotteries Tennessee 3 Teachers Schools Random STAR Double-Dose 2 Children Schools Random Algebra

Theoretical Model Within Sites     2 , ~ ( 0 , ) Y U B T r r 0 ij j j ij ij ij T Between Sites     2 , ~ ( 0 , ) U u u 0 0 0 0 0 j j j     2 , ~ ( 0 , ) B b b j j j b   ( , ) Cov u b 0 0 j j b

4. Unbalanced Designs and Targets of Inference Two kinds of imbalance • Unequal n per site • Unequal propensity score (probability of assignment to the treatment group)

Targets of Inference Generalize to a Populati on of Sites * 1 J    B * sites j J  1 j * 1 J      2 2 ( ) B * b sites j J  1 j * 1 J        ( )( ) U B 0 0 0 * b sites j j J  1 j Generalize to a population of Persons * * J J     / N B N persons j j j   1 1 j j * * J J       2 2 ( ) / N B N b persons j j j   1 1 j j * * J J         ( )( ) / N U B N 0 0 0 b persons j j j j   1 1 j j

Identification 1 2 Combine level and level        Y T u b T e 0 0 ij ij j j ij ij       ( | ) ( | ) ( | ) E Y T T E u T T E b T 0 0 ij ij ij j ij ij j ij ( | . ), ( | . ) Worry about E u T E b T 0 j j j j e.g., charter school lotteries

Parameters to be Properties estimated a)Fixed effects β Biased if precision related to B b)Centering HLM B,Var(B) Bias in β if precision is related to B (less so than fixed effects ) c) Control B,Var(B)?? Similar to Centering for β , propensity Score d)Weighting B,Var(B),Cov(B,U 0 ) Removes bias (“IPTW”) with (but may be imprecise!) HLM

a) Fixed Effects Model      Y T e ij ij j ij J  ˆ  . ( 1 . ) n T T B j j j j ˆ    1 j J   . ( 1 . ) n T T j j j  1 j where ˆ   B Y Y 1 0 j j j

Potential Bias if heterogeneous impact, and unbalanced design   J   . ( 1 . ) n T T B    j j j j  ˆ    1 ( | , ) j   E B T E J     . ( 1 . ) n T T   j j j    1 j  A precision weighted average

b) Centering the Predictor AND the Outcome (“ FIRC ” Model)     ( . ) Y Y B T T e ij j j ij j ij     2 ~ ( 0 , ) B b b N j j j b

Robustness (compared to fixed effects)    1    2 J      2   B   b . ( 1 . ) j  n T T   ˆ  1      j 2 2 j j j ( | , , , )   E T E  1 b      2 J    2      b . ( 1 . ) n T T      1 j j j j  2 , , As heterogene ity increases increases reliance on b  . ( 1 . ) . n T T decreases j j j

Estimation of heterogeneity  2     2 ( 1 ) m J  ˆ        ( 1 ) 2 ( 1 ) 2 [( ) ] m m   B V  b j j . ( 1 . ) n T T    1 j  ( ) 2  j j j m  2 b     2 ( 1 ) m J     ( 1 ) 2 m    b . ( 1 . ) n T T    1 j j j j  2 , , As heterogene ity increases increases reliance on b  . ( 1 . ) . n T T decreases j j j

c) Control the Propensity Score 1 Level    Y U B T e 0 ij j j ij ij 2 Level      . U T u 0 0 0 j j j    B b j j , Very similar to centering for average treatment effect  2 no good estimate of b

Variance Estimation    1       J    1 1     ( )  ( 1 )  ( 1 )  ( 1 )  ( 1 ) ( τ ) τ V τ V m  m m m m  vec j j    1 j       J    1 1          ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) * τ V τ V ( d d ) m m m m m m T vec j j j j  1 j ˆ         ( 1 ) ( 1 ) m m U T    0 0 1 j j d   ˆ    ( 1 ) j m B   j

Summary so far Four commonly used options * limit what we can estimate * produce inconsistent estimates of what we can estimate.

6.HLM with Inverse Probability of Treatment Weighting • IPTW • Embedding within HLM • Always produces consistent estimates • May be quite imprecise

Inverse Probability of Treatment Weights (Robins and Greenland, 2000)   1 , / If T w T T ij ij j     0 , ( 1 ) /( 1 ) If T w T T ij ij j so  1 T T    ( 1 ) w T T  ij ij ij 1 T T j j

Weighting Schemes for HLM Level-2 Level-1 weight Result weight Weights site-specific estimates of impact by   n ( 1 )( 1 ) The picture can't be displayed. n j T T T T j  ij ij  . 1 . n T T j j Weights site-specific     ( 1 )( 1 ) T T T T estimates equally n    1 ij ij    . 1 . n T T   j j j

Estimation via Weighted log Likelihood (weighted complete-data log likelihood)       2 , ~ ( 0 , ), ~ ( 0 , ) T T Y X Z u e u N e N j ij ij j ij j ij   ln[ ( , )] ln[ ( | )] ln[ ( )] h Y u f Y u p u : After weighting n n J J  j  j          2 2 2 1 ( T T ) / log( 2 ) l w Y X Z u N 2 w ij ij ij ij rj ij     1 1 1 1 j i j i J     1    1 log( 2 | |) T w u u J 2 2 j j j  1 j n J J j     , where w N w J 2 ij j    1 1 1 j i j

Two examples of weighting schemes Equally weight people Equally weight sites Level-1 weight Level-1 weight    1  1 T T T T       ( 1 ) ( 1 ) n T T T T    ij ij  n 1 ij ij 1 T T j   T T j j j j Level-2 weight Level-2 weight 1 n j / n J J J    ˆ ˆ ˆ ˆ     1 / / n B n B J j j j j   1 1  j j 1 j     J J   ˆ ˆ ˆ ˆ             ˆ 2 1 2 2 1 n 2 ( ) ˆ ( ) J B V j J B V b j j b n j j  1 j  1 j     J J   ˆ ˆ ˆ ˆ ˆ ˆ          1 n   1      ˆ ( )( ˆ ) ˆ ( )( ˆ ) J j B U C J B U C 0 0 0 0 0 0 b n j j j b j j j   1 1 j j

7. Discussion A Class of Estimators … We are approximating various balanced designs by maximizing the weighted log likelihood This gives us a family of method-of-moments estimators Hence no reliance on normality or homoscedasticity May be quite imprecise

Extensions • Extend easily to multi-site clustered randomized designs • Balance propensity within covariate classes in randomized studies • Extend to observational studies • Extend to weighted mediation models (Hong et al, 2012) • Application to local average treatment effects (Raudenbush, Nomi, and Reardon, 2016)