Methodological Challenges and Innovations for Testing Multilevel Causal Theory in Implementation Science Nate Williams, PhD Steve Marcus, PhD Boise State University University of Pennsylvania School of Social Work School of Social Policy & Practice 1 WWW.UPENN.EDU
Mediation Analysis c X Y Does variation in Does variation in M explain variation X explain variation in Y? in M? M a b c' X Y 2 WWW.UPENN.EDU
Tests for Mediation (Hayes & Scharkow, 2013; MacKinnon et al., 2001, MacKinnon et al., 2004) Test Pros Cons • • Baron & Kenny causal steps Relative ease of implementation Low power • • Doesn’t estimate or test approach Familiar to reviewers parameter of interest • No effect size • • Doesn’t estimate or test the Joint significance test Good power • Ease of implementation parameter of interest • • NHST approach No effect size • • Product of coefficients Ease of implementation Suboptimal power • • approach with Sobel test Can produce an effect size Inaccurate normality assumption • Familiar to reviewers • • Doesn’t provide p -value Product of coefficients Optimal power • • approach with asymmetric Ease of implementation Not an NHST approach • 95% CIs Excellent Type I error protection • Can produce an effect size • Directly tests parameter of interest 3 WWW.UPENN.EDU
Product of Coefficients Approach a*b = ??? 1. Run a series of regression models Asymmetric 95% CI = ?? to ?? that estimate the paths of interest Joint Significance Test = ?? 2. Multiply coefficients to obtain an P m = ?? estimate of the indirect effect(s) 3. Test the statistical significance of the M indirect effect(s) using the Joint a b Significance Test and develop Asymmetric 95% Confidence Intervals c' X Y 4. Generate measures of effect size 4 WWW.UPENN.EDU
Multilevel Data Creates Problems for Standard Mediation Analysis (Krull & MacKinnon, 2001; Preacher, 2015) Agencies 1. Nesting creates non-independence of observations which violates the assumptions of standard statistical models Clinicians Patients WWW.UPENN.EDU
Multilevel Theory Creates Problems for Standard Mediation Analysis 2. Multilevel theory is not Implementation easily incorporated into Federal Leadership and standard statistical State Climate models District • X, M, & Y might Attitudes reside at different School levels Beliefs Norms Intentions Intensity • Variation in the outcome is assumed to occur at multiple Knowledge levels Self-Efficacy Skill 6 WWW.UPENN.EDU
Multilevel Modeling as a Partial Solution • MLM allows investigators to treat variation in the outcome that occurs across clusters as a phenomena of interest rather than a nuisance • MLM permits the inclusion of predictor variables at multiple levels • Relative to a fixed effects approach, MLM allows researchers to account for the non- independence of nested observations in a way that is both parsimonious and flexible 7 WWW.UPENN.EDU
Using MLM to Conduct Mediation Analyses • The product of coefficients approach can be easily extended to multilevel mediation models (Krull & MacKinnon, 2001; Zhang et al., 2009) a Organizational Organizational Level 2 (Organization) antecedent Mediator b Level 1 (Individual) c' Clinician outcome Organizational Level 2 (Organization) antecedent c' a Level 1 (Individual) b Clinician Clinician mediator outcome 8 WWW.UPENN.EDU
The Problem of Conflated Slopes in Multilevel Mediation Analysis • Level 1 variables have variance at level 1 (within groups) and level 2 (between groups) 9 WWW.UPENN.EDU
Two Potentially Different Relationships between X and Y • Because a level 1 X and a level 1 Y have unique variances at Level 1 and Level 2, their relationship to each other can differ across levels • Estimating the X – Y relationship using a single slope, as is sometimes recommended for multilevel mediation analyses, “conflates” the two relationships and leads to biased parameter estimates and incorrect statistical tests in mediation analysis (Zhang et al., 2009; Enders & Tofighi, 2007; Kreft et al., 1995) 10 WWW.UPENN.EDU
Un-conflating Slopes in Multilevel Mediation Analysis Centered within Context with Means Reintroduced Approach – CWCM (Zhang et al., 2009) • Two new variables are created which partition the variance in M into two parts: - group means (𝑁 𝒌 ) - group-mean centered scores 𝑁 𝑗𝑘 − 𝑁 𝒌 • Both variables are used in the analysis but only one variable (group means) is used to calculate the indirect effect 11 WWW.UPENN.EDU
Example of Un-conflated MLM Mediation Analysis a*b = .035 Organizational culture c' Asymmetric 95% CI = .011 to .067 a Joint Significance Test = Sig. Clinician b Clinician behavior intentions P m = .62 (use of EBP) 12 WWW.UPENN.EDU
Conclusions • Standard analytic approaches to mediation analysis are often insufficient for hypothesis testing in implementation science because of nested data and multilevel theory • Multilevel modeling is a useful tool for addressing these issues but problems can arise if investigators ignore the “conflated slopes issue” • Simple procedures are available that allow investigators to un-conflate slopes in MLM and generate accurate estimates of indirect effects 13 WWW.UPENN.EDU
THANK YOU! Nate Williams, PhD natewilliams@boisestate.edu Steve Marcus, PhD marcuss@upenn.edu 14 WWW.UPENN.EDU
References Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: a new look at an old issue. Psychological Methods, 12(2), 121-138. Hayes, A. F., & Scharkow, M. (2013). The Relative Trustworthiness of Inferential Tests of the Indirect Effect in Statistical Mediation Analysis: Does Method Really Matter?. Psychological Science (0956- 7976) , 24 (10), 1918-1927. Kreft, I. G. G., de Leeuw, J., & Aiken, L. S. (1995). The effects of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30, 1-21. Krull, J. L., & MacKinnon, D. P. (2001). Multilevel Modeling of Individual and Group Level Mediated Effects. Multivariate Behavioral Research , 36 (2), 249-277. doi:10.1207/S15327906MBR3602_06 MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods , 7 (1), 83- 104. 15 WWW.UPENN.EDU
References Mackinnon, D. P., Lockwood, C. M., & Williams, J. (2004). Confidence Limits for the Indirect Effect: Distribution of the Product and Resampling Methods. Multivariate Behavioral Research , 39 (1), 99. Preacher, Kristopher J. 2015. Advances in mediation analysis: a survey and synthesis of new developments." Annual Review Of Psychology 66, 825-852. Preacher, K. J., & Kelley, K. (2011). Effect size measures for mediation models: quantitative strategies for communicating indirect effects. Psychological Methods , 16 (2), 93-115. doi:10.1037/a0022658 Zhang, Z., Zyphur, M. J., & Preacher, K. J. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods , 12 (4), 695-719. doi:10.1177/1094428108327450 16 WWW.UPENN.EDU
Recommend
More recommend
