Measuring Year-to-Year Improvement in Risk-Adjusted Outcome Measures: Filling a Methods Gap Academy Health ARM June 27, 2016 1
Funding and Disclosures • This work was funded by a contract with the Centers for Medicare & Medicaid Services, Center for Clinical Standards and Quality • CMS Government Task Leader: Vinitha Meyyur • Contract #: HHSM-500-2013-13018I, Task Order HHSM-500-T0002 – Modification 0002 2
Outline • Need for improvement measurement methods • Goals for methods • Case Study: ACO admission measure in Medicare Shared Savings Program (SSP) – Program design, goals, challenges – Three methods – Results – Evaluation methods against program and technical goals 3
Why Methods for Measuring Improvement? • Providers want credit for improvement • Comparing year-over-year risk-adjusted scores shows change in relative rank • Programs increasingly incorporating improvement scores in payment models 4
Goals for Methods • Fit with program design and goal • Technically valid and feasible • Practical to implement • Usable for program and providers – Detect meaningful improvement? – Are readily understood? – Support target setting? 5
Case Study: ACO RISK-STANDARDIZED ACUTE ADMISSION RATES AMONG PATIENTS WITH HEART FAILURE 6
Medicare SSP Improvement Bonus • ACOs earn quality points on a sliding scale based on level of performance in four quality domains • Shared savings based on quality points • ACOs can earn bonus improvement points for statistically significant improvement from one year to the next: • Size of improvement does not matter • ACOs who reach 90 th percentile or better earn maximum, regardless of improvement • Getting worse on one measure cancels out improvement in another for measures in same domain => two-sided test 7
ACO Measure Tested: ACO 37 – Risk-Standardized Acute Admission Rate among Heart Failure Patients Cohort: • Non-hospitalized heart failure patients Medicare FFS age 65+ • Assigned to ACO Outcome • Number of acute, unplanned hospital admissions per 100 person-years at risk for hospitalization Data • Medicare Claims Risk-adjustment model • Hierarchical negative binomial model • Risk-adjustment variables: • Age • 21 comorbidities • Cardiac device variable 8
Measurement Challenges • An ACO’s patients change from Year 1 to Year 2 • Admission risk of enrollees changes from Year 1 to Year 2 • Natural events may affect admission rate • Regression to the mean could contribute to year-to-year change 9
Population Studied • Patients: Medicare FFS patients 65+ • Year 1 (2012)= 123,626 • Year 2 (2013)= 134,961 • ACOs • 114 ACOs in SSP in both 2012 and 2013 • ACO volume range: 303 - 9,914 (median: 690) • ACO risk factor frequencies in Y1, Y2 similar 10
Two-Thirds of Patients Stay in Their ACO Distribution of the percentage of Y1 patients also in Y2 across ACOs (79,942 of 123,626 patients stay in their ACO from Y1 (2012) to Y2 (2013) [64.7%]) 11
Overall Variables (N=258,587) Y1 Y2 Age Mean(std) 79.8 (7.8) 79.8 (7.8) Race White 87% 87% Black 9% 9% Male 51% 51% High risk cardiovascular (CV) factors 32.8% 31.9% Low risk CV factors 85.4% 84.1% Arrhythmia 64.4% 63.3% Structural Heart Disease 41.5% 39.9% Advanced cancer 7.8% 7.6% Dementia 22.4% 21.3% Diabetes w/ complications 52.6% 52.2% Dialysis 3.1% 3.1% Disability/Frailty 22.9% 21.8% GI/GU 33% 32% Hematology 17% 15% Infection & immune disorders 6% 7% Kidney disease 39% 39% Liver disease 2% 2% Neurological 45% 44% Psychiatric illness/Substance abuse 37% 36% Pulmonary disease 59% 57% Other advanced organ failure 20% 19% CRT/ICD/Pacemaker 24% 23% Iron deficiency anemia 54% 52% Major organ transplant 0% 0% 12 Other organ transplant 1% 0%
Tailored Objectives for Method • Fits with program design • Compares ACO to itself (not used to compare quality among ACOs) • Accounts for enrollee and risk shifts • Technically feasible • Statistically tests for better or worse rate • Detects improvement • Readily implemented • Useable by ACOs • Provides target rate 13
ACO methods tested Options Year 1 as Pre-Post Matched Benchmark Test Patients Objectives Fits with program goal: ACO vs. self Addresses risk shifts Addresses regression to mean Identifies statistically significant change Readily implemented Usable 14
Year 1 Benchmark • Fit risk model to each ACO in Year 1 • Apply model coefficients to Year 2 data to calculate expected for Method Year 2 • Observed (Year 2) – Expected (Year 2) Score • Significance level: 0.05 Results for 114 ACOs 15
ACO methods tested Options Year 1 as Pre-Post Matched Benchmark Test Patients Objectives Fits with program goal: ACO vs. self Addresses risk shifts Addresses regression to mean Identifies statistically significant change Readily implemented Usable 16
Option 2: Pre-Post Test • Fit risk model to ACO using Year 1 & Year 2 combined • Add a year variable (Y2=1; Y1=0) Method • Exponentiate year variable to get rate ratio (<1= improvement) • Score Significance level: 0.05 Results for 114 ACOs 17
ACO Methods Tested Options Year 1 as Pre-Post Matched Benchmark Test Patients Objectives Fits with program goal: ACO vs. self Addresses risk shifts Addresses regression to mean Identifies statistically significant change Readily Implemented Usable 18
Option 3: Matched Patients • Use propensity score matching to match patients from Year 1 and Year 2 (Mahalanobis distance matching method) • Fit risk model to subset of matched patients adding a year variable Method (Y2=1; Y1=0) • Exponentiate year variable to get rate ratio (<1=improvement) Score • Significance level 0.05 Proportion of Year 2 matched patients: 83%-98.8% Results • Tested in 12 ACOs, 3 in each of 4 volume quartiles for 12 • 1 in highest volume quartile significantly improved ACOs • 11 showed no statistically significant change 19
ACO methods tested Options Set Year 1 as Pre-Post Matched Benchmark Test Patients Objectives Fits with program goal: ACO vs. self Addresses risk shifts Addresses regression to mean Identifies statistically significant change Readily implemented Usable 20
ACO Methods Tested Options Year 1 as Pre-Post Test Matched Benchmark Patients Objectives Fits with program goal: ACO vs. self Addresses risk shifts Confident about risk-adjustment Addresses regression to mean More stable model coefficients Identifies statistically significant change Readily Implemented Usable Can set target early in Year 2 21
Assessment Similar for the 114 ACOs: Year 1 Benchmark vs. Pre-Post Test Pre-Post Test Year 1 Option 1 Significantly Significantly total Benchmark No different better worse 17 (14.9%) 13 (11.4%) 0 (0.0%) Significantly better 30 (26.3%) No different 0 (0.0%) 81 (71.1%) 0 (0.0%) 81 (71.1%) Significantly worse 0 (0.0%) 2 (1.7%) 1 (0.9%) 3 (2.6%) 114 (100%) Option 2 total 17 (14.9%) 96 (84.2%) 1 (0.9%) *Concordance: 86.9%; Kappa=0.64 22
And similar for 12 ACOs evaluated with Patient Matching Performance Status 12 ACOs Year 1 Benchmark Pre-Post Test Patient Matching Volume quartile 1 (low volume): 1 No different No different No different 2 No different No different No different 3 Significantly improved Significantly improved No different Volume quartile 2: 4 No different No different No different 5 No different No different No different 6 No different No different No different Volume quartile 3: 7 No different No different No different 8 No different No different No different 9 No different No different No different Volume quartile 4 (high volume): 10 No different No different No different 11 Significantly improved Significantly improved Significantly improved 12 No different No different No different 23
Summary and Conclusion • Criteria for method should be aligned with program goals • All methods were technically feasible • The three methods have different strengths and weaknesses • The results were aligned • The choice should reflect program priorities 24
Acknowledgements • Craig S. Parzynski, M.S. • Haikun Bao, Ph.D • Jeph Herrin, Ph.D. • Faseeha K. Altaf, M.P.H. • Hayley J. Dykhoff, B.A. • Mayur Desai, Ph.D., M.P.H. • Susannah M. Bernheim, M.D., M.H.S. • Zhenqiu Lin, Ph.D. 25
QUESTIONS? 26
Recommend
More recommend
