Balancing Factors for Stepped Wedge Designs Robert Lew 1 Hongsheng Wu - - PowerPoint PPT Presentation

Balancing Factors for Stepped Wedge Designs

Robert Lew 1 Hongsheng Wu 1,2 , Christopher Miller 1,3 , Bo Kim 1,3 Mark Bauer 1,3

1VA Boston Healthcare System, 2Wentworth Institute of

Technology, 3Harvard Medical School

Design of BHIP-CCM trial

• This trial has a pre-post design that will implement a team-
• riented psychiatric patient management system at 9 sites

(Veterans Affairs hospitals), 3 sites per period over 3 periods.

• We might assign sites ABCDEFGHI to periods as follows:

Period 1 Period 2 Period 3 A B C D E F G H I

Time trends: what could possibly go wrong?

• Suppose each of 9 sites is either Urban (U) or Rural (R).
• Permute sites to Balance both U and R across the periods

Period 1 Period 2 Period 3 A B C U U U D E F R R R G H I R R R Period 1 Period 2 Period 3 A D G U R R B E F U R R C H I U R R

Imbalance score for Times -1, 0, and 1

• Average time for U is -3/3 = -1; for R is (0+0+0+1+1+1)/6 = 0.5.
• The average time for both U and R across the periods is zero.

Period -1 Period 0 Period 1 A B C U U U D E F R R R G H I R R R Period -1 Period 0 Period 1 A D G U R R B E F U R R C H I U R R

Try many site permutations and pick best

• Of 20000 ABCDEFGHI permutations, only 1680 were

distinct.

• Restricted selection to the 34 best balanced site assignments

(restricted randomization).

• ‘Best’ means the least imbalance score overall; the sum for

every absolute category imbalance score for every factor.

• Randomly chose 1 of these 34 best to mute time-trend bias.

Minimize the Overall Imbalance Score

• We describe results for only 4 of many factors (characteristics):
• Urban (U-urban, R-rural),

Academic affiliate (yes, no),

• Staff experience (high, low),

Region of USA (A, B, C, D)

• Each factor has several categories: 10 categories listed above.
• We must balance each of the categories over the 3 periods.
• Perfect balance is zero: a positive score shows imbalance.
• Overall score sums absolute imbalance over all categories.

Category N =1680 N=34 Category N=1680 N=34 𝒏𝒇𝒃𝒐 ± 𝑡𝑒 𝒏𝒇𝒃𝒐 ± 𝑡𝑒 𝒏𝒇𝒃𝒐 ± 𝑡𝑒 𝒏𝒇𝒃𝒐 ± 𝑡𝑒 Urban 0.20 0.16 0.12 0.13 Region A 0.25 0.21 0.14 0.13 Rural 0.25 0.20 0.15 0.16 Region B 0.41 0.35 0.14 0.23 Academic 0.25 0.21 0.10 0.14 Region C 0.41 0.35 0.11 0.21 Non_acad’c 0.20 0.17 0.08 0.11 Region D 0.66 0.47 0.04 0.20 High %Staff 0.16 0.13 0.03 0.07 Low %Staff 0.31 0.26 0.07 0.13 Overall 3.10 0.91 0.99 0.32

Category scores for 4 Factors and Overall Score.

Related task: construct a “comparable” control group to our 9 sites that has face validity.

• Minimize ‘distance’ between study and control with respect to 10 factors:
• AES Psychological Safety AES Civility AES QI/SR %Rural Veterans

#Psychiatric Teams #Patients % PACT15 Patients #High risk patients #Phone Encounters Geographic region

• Problem:

Ad hoc solution:

• weight, rescale factors? Equal weight and stdev = 1
• Delete redundant factors? Pearson r > 0.9
• Use numbers or tertile categories? Numbers
• What distance metric? Absolute difference
• Region of US

MATCH

Chose 27 ‘control’ sites to compare with 9 study sites matched on 10 factors

• VA Geographic Regions are large and irregular.
• Potential Report on Patient Mental Health Symptom Score

Our 9 sites 27 comparable sites All VA but our 9 Mean score 13.2 15.3 19.6

• Several control groups may clarify the apparent result.
• 1. NY

Virginia, Virginia, NY

• 2. NY

Pennsylvania, MD, RI

• 3. NY

Pennsylvania, NC, NC

• 4. KY

Florida, KY, SC

• 5. Ohio

Illinois, Iowa, Iowa

• 6. Ohio

Ohio, WI, Minnesota

• 7. Kansas

Minn, Missouri, Kansas

• 8. Texas

Miss, Montana, Ariz

• 9. Texas

Louisiana, Ariz, Utah

Control (referent) group construction when we cannot randomize (enough)

• Time trend example drew on ANOVA balanced incomplete block

designs. Basic underlying statistical model for stepped wedge designs.

• Matching/balance controls example drew on case/control studies.

These catalog why many simple methods fail and offer remedies.

• Observational study design: Rubin/Rosenbaum propensity scores.

Propensity matching often fails and makes matters worse.

• Causal models. Factors may play different roles and are not merely

simple covariates in a prediction model.

CONCLUSION: Protecting against bias relies heavily on the wisdom of context experts.

• What factors matter?
• Try to have at least surrogates for all major factors.
• Choose assignment or a control group that has face validity.
• Start with conceptually simple criteria for comparability.
• Only use fancy concepts (propensity) understanding the limitations.
• Construct several control groups.
• Check that comparability is robust against different choices for factor

categories, factor weights, cutpoints, and other subjective choices. THANK YOU!