Non-linear Difference-in-Differences Models for Policy and Program Evaluation Claude M. Setodji, Justin W. Timbie, Rosalie Malsberger, Katherine Giuriceo, Katherine L. Kahn June 25, 2017 Supported by CMS Contract #HHSM-500-2005-00028I Task Order #T0008. The contents of this presentation are solely the responsibility of the authors and do not necessarily represent the official views of the U.S. Department of Health and Human Services or any of its agencies.
Outline Challenges associated with non-linear difference-in-differences (DID) models Puhani’s non-linear DID estimator Simulated example Estimating effect of patient-centered medical home recognition on expenditures (modeled using log transformation) Recommendations 2
Difference-in-Differences Methods are a Popular Choice for Program Evaluation In a DID design, outcomes are assessed both before and after program implementation in both an intervention and control group Comparisons are made between trends in both groups with the interaction between group and time being the parameter of interest in the model DID is commonly used because of its strengths: Mimics random assignment for baseline similarity Allows estimation of causal inference Controls for unmeasured confounders making it more powerful than cross-sectional studies 3
Non-linear Models Produce Estimates That Are More Difficult to Interpret Non-linear models require transformations to achieve normality Log transformations are often used when modeling expenditures, to account for skewed distributions Estimates of program impacts are on the scale of log dollars, which are difficult to interpret “Recycled prediction” or marginal estimates can express estimates on a meaningful scale 4
Recycled Prediction Uses Model Estimates to Set Up Two Counterfactuals Two Counterfactual Questions What if all patients received the intervention? What if no patients received the intervention? Methods Expenditures are estimated under both scenarios assuming model estimates are correct, after back transformation from log The difference is a good proxy for the intervention’s main effect estimate However, in DID context, the difference representing the interaction can be misleading 5
Naïve Non-Linear DID Estimator In the DID model, there are 4 cells (1) Baseline, Intervention (2) Baseline, Control (3) Follow up, Intervention (4) Follow up, Control A simple approach would generate recycled predictions for each cell, and then estimate: Change in Intervention from baseline to follow up Change in Control from baseline to follow up DID Estimate = Difference of the two differences 6
Naïve Estimator is Misleading for Non-Linear Models The Problem: The log of a difference is not the difference of the logs Monotonicity is lost with log transformation The model coefficients are correct, but the recycled predictions are not Proposed solution: The Puhani solution Puhani uses the interaction as a single variable Use it like a predictor variable that can be turned on and off for recycling and estimating differences 7
Puhani DID Estimator Uses two recycled predictions: Interaction “turned on.” Set everyone to have interaction value 1 Interaction “turned off.” Set everyone to have interaction value 0 Take the difference in prediction as a representation of the interaction estimate Can be estimated using margins command in Stata 8
Simulated Example of Two DID Estimators We use simulated data from the evaluation of CMS’s FQHC Advanced Primary Care Practice Demonstration to show an extreme case, to assess how Patient-Centered Medical Home (PCMH) recognition affects annual expenditures Simulated data were modeled after Part A and B fee-for-service Medicare claims data 428,146 adult Medicare beneficiaries with ≥1 visit to a national sample of 1330 FQHC clinics Beneficiary outcomes and PCMH recognition were simulated from patient and clinic characteristics Hypothesis: Recognition will lead to decreased expenditures 9
Beneficiary Characteristics in Simulated Sample Clinics with Clinics without PCMH Recognition PCMH Recognition Age (in years) 18-64 45.5 44.1 65-74 34.2 34.7 75-84 15.5 16.0 85 4.8 5.2 Race/Ethnicity White 69.1 69.6 Black 17.2 18.4 Hispanic 6.4 7.1 Other/ Unknown 7.2 4.8 Male 54.5 54.0 Disabled 52.3 51.2 10
Skewed Expenditure Needs Transformation Log transformation of Expenditure is skewed expenditure returned a to the right, requiring normal distribution outcome transformation 11
Log Scale Results Log scale P-value PCMH Recognition 0.926 0.000 Post 0.922 0.000 PCMH recognition x Post -0.227 0.000 Age 0.005 0.000 Race: Black 0.103 0.012 Hispanic 0.067 0.112 Other/ Unknown 0.170 0.001 Male 0.033 0.183 FQHC clinics with PCMH recognition have smaller increases in expenditures than non-recognized clinics Recognition reduces expenditures by -0.227 log dollars 12
Naïve Recycled Prediction Result Before and After Baseline Follow up Difference No PCMH $186.03 $467.73 $281.70 Recognition [163.19, 208.81] [410.58, 524.92] [245.84, 317.61] PCMH $469.79 $941.60 $471.81 Recognition [411.09, 528.51] [823.82, 1059.38] [405.77, 537.72] Difference $283.76 $473.87 + $190.11 [244.18, 323.31] [402.90, 544.79] [145.78, 234.32] • The interaction coefficient was -0.227 while the “naïve” recycled estimate is $190.11 • Both are statistically significant; but one is positive one is negative 13
Puhani’s Recycled Prediction Result The interaction is effectively modeled as a binary variable the same way we use intervention/control Recycled counterfactual framework Estimate 95% CI Interaction = 0 $525.92 [461.39, 590.41] Interaction = 1 $419.23 [366.80, 471.72] Difference -$106.69 [-135.35, -78.03] PCMH recognition is associated with a reduction in expenditures of $106.69 The interaction coefficient was -0.227 14
Summary and Recommendations Policymakers require easy to interpret estimates of program impacts — especially from non-linear models Recycled predictions involving interactions can produce misleading conclusions unless appropriate estimators are used The Puhani estimator can be used with any non-linear monotonic transformation and can be implemented easily in SAS or STATA 15
16
Recommend
More recommend
