How to Predict Overdose Death with PDMP Data and Advanced Analytics Live Webinar Thursday, March 16, 2017
Sp Spon onsored by: y: Live Webinar 3/16/17 | 1:00 p.m. ET
Q+A – Submit a question, located below the slides Resources List – Access website links and download slides Help – Submit any technical issues, located below the slides Live Webinar 3/16/17 | 1:00 p.m. ET
Twitter Join the discussion on Twitter! Live tweet using the hashtag #RXLiveWebinar Live Webinar 3/16/17 | 1:00 p.m. ET
How to Predict Overdose Death with PDMP Data and Advanced Analytics A cooperative effort between OARRS and Appriss Health
Speakers Chad Garner, MS • OARRS Director Jim Huizenga, MD • Emergency Physician, BCEM • Chief Clinical Officer for Appriss Health David Speights, PhD • Ph.D. Biostatistics • Chief Data Scientist for Appriss
Disclaimer • Dr. Huizenga and Dr. Speights are both employees of Appriss Health. • Mr. Garner is the PMP director for the Ohio Automated Rx Reporting Service (OARRS). • He certifies that he has NO affiliations with or involvement in any organization or entity with any financial interest (such as honoraria; educational grants; participation in speakers’ bureaus; membership, employment, consultancies, stock ownership, or other equity interest; and expert testimony or patent-licensing arrangements), or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript.
Objectives • Explain how PDMP data and advanced analytics can impact detection of unintentional overdose deaths • Identify comprehensive data results, from a variety of complex data patterns, for early detection of overdose risk • Review the early identification process of prevention and management of substance use disorders in the U.S. and OARRS
Overview • Data Overview • Initial Analysis • Secondary Analysis • Advanced Data Analysis • Summary and Future Directions
Perspective “Treatment without prevention is simply unsustainable” Bill Gates
Data Overview OARRS Operational Overview • Operational since October 2, 2006 • Collects information regarding all Schedule II-V controlled substances dispensed by Ohio-licensed pharmacies • 25 million prescriptions, 4 million patients per year • Patient-identifiable data kept for 3 years • Patient-deidentified data kept indefinitely
Data Overview - Decedents 2014 Death Data • Data Sample • 2,482 unintentional overdose deaths • Inclusion Criteria • Department of Health determined death due to unintentional overdose between Jan 1 – Dec 31, 2014 • Matching ID within OARRS • Overdose death data not available until June 2015 • OARRS keeps 3 years of patient-identifiable data • Selection results • 1,687 decedents
Initial OARRS Analysis Microsoft SQL Server Analysis Services was used to create predictive data models across 12 data measures using 4 different algorithms. 1. Microsoft Decision Trees* 2. Microsoft Clustering 3. Microsoft Naïve Bayes 4. Microsoft Logistics Output was used to trim the data model to 4 data measures showing strong association with overdose death.
Initial Analysis Four risk factors strongly associated with OD death Risk Factor Odds Ratio Pharmacies ≥ 4 3.7 Benzo / Opioid overlap ≥ 35 days 2.4 Max MED ≥ 100 2.3 Cash Payment ≥ 1 2.2 All 4 present 10.3
Odds and Odds Ratios What are odds? • Chance of an event occurring divided by the chance that the event won’t occur • If the chance of something is small then it is approximately equal to the probability What are odds ratios? • Odds ratios are a ratio which compares one group to another group and is used to express relative risk
Odds and Odds Ratios - Example Example Group Chance Of Outcome Odds 1 0.07% 0.07% / 99.93% = .07% 2 1.50% 1.50% / 98.50% = 1.52% Odds Ratio of group 2 compared to group 1 1.52 / 0.07 = 21.71 Group 2 is more than 20 times as likely to suffer the outcome as Group 1
Initial Analysis Four risk factors strongly associated with OD death Risk Factor Odds Ratio Pharmacies ≥ 4 3.7 Benzo / Opioid overlap ≥ 35 days 2.4 Max MED ≥ 100 2.3 Cash Payment ≥ 1 2.2 All 4 present 10.3
Secondary Analysis Using Narx Scores as a predictor of overdose death • Type specific use indicators for narcotics, sedatives and stimulants • Range from 000-999 • As the score increases, so does the presence of: • Providers • Pharmacies • MME • Overlaps
Narx Scores in vivo
Narx Scores in vivo Narx Scores In Workflow
Narx Scores Narx Score Distribution Scores < 200 200-499 500-650 >650
Narx Scores Summary Narx Scores are • Numerical representations of PDMP data that capture “use”. • Information at a glance. Narx Scores are not • Rules (they are tools). • Are not synonymous with abuse.
Secondary Analysis Methodology – Narx Scores as a predictor of overdose risk • 100:1 case / control study • Determined the highest narcotic score in the year prior to death for each decedent and for 100 date-matched living controls • Calculated Odds Ratios
Secondary Analysis Results Narcotic Score Odds Ratio 0-199 1 200-299 6.4 300-399 7.4 400-499 10.2 500-599 15.5 600-699 23.3 700-799 29.8 800-899 37.7 900-999 63.4
Primary and Secondary Analysis Summary Using traditional techniques that include both red flags and a composite use indicator, we were able to determine significant associations with unintentional overdose death • Initial analysis identified 4 red flags strongly associated with overdose death risk • Secondary analysis strongly associated Narx Scores with overdose death risk
Machine Learning Approach Data Overview Used the same data from the secondary analysis • 1:100 case to controls • Artificial resultant 1% incidence of disease Applied machine learning and other predictive techniques to develop a 3-digit score similar to Narx Scores, termed an Overdose Risk Score • Range from 000-999 • Risk doubles approximately every 100 points • Similar distribution to Narx Scores
Advanced Data Analysis – General Method Data Inputs Prescriber/Pharmacy Visit History and Acceleration Drug Dispensation History and Trends Decision Engine Types of Machine Learning Narcotics/Sedatives Overdose Risk Models Optimized Score by Simulated (000 to 999) Narcotics to Morphine Annealing Equivalencies Inputs processed through Literature Defined Red Flags predictive models to determine the composite risk High Risk Behavioral Patterns
Machine Learning Approach Variable Derivation Variable Determination • Hypothesize a variable and the expected effect • Develop variable for case and controls • Determine independent predictive ability More than 70 variables were evaluated using this approach Examples • Amount of narcotics (in MME) used in the prior 365 days • Amount of sedatives (in MME) used in the prior 60 days From the 70 variables, approximately one dozen chosen for final model • Some that are used for Narx Scores • Some that are used for Red Flags • Some new variables that look at change over time
Machine Learning Approach Variable Derivation Model Validation During Development • For each decedent and matched control 4 random dates were chosen in the one year prior to the date of death for the decedent producing 4 separate modeling sets to use in model fitting and evaluation. • Each Set was further split into a 75% training sample and 25% validation sample Final Model Validation • After model completion, we used death data from 2013 and 2015 to validate the final model and compare to the 2014 results
Machine Learning Approach Model Evaluation with the KS Statistic • Kolmogorov-Smirnov (KS) Statistic measures the maximum difference between the cumulative percentage of two populations ( Non-Decedents vs Decedents ) by score. • Standard metric used in statistics to evaluate models. 100% Non-Decedents CUMULATIVE % KS Decedents 0% High Low SCORE
Model Score Distribution for Decedents and Non-Decedents Model Scores for Decedents and Non-Decedents Non-Decedents Decedents 40% 35% Percent of Population Decedents have 30% higher risk scores 25% Non-decedents Decedents 20% Mean: 209 Mean: 505 Median: 209 Median: 507 95 th %tile: 95 th %tile: 569 835 99 th %tile: 99 th %tile: 15% 730 938 10% 5% 0% 0 to 50 51 to 101 to 151 to 201 to 251 to 301 to 351 to 401 to 451 to 501 to 550 to 600 to 650 to 700 to 750 to 800 to 850 to 900 to 950 to 100 150 200 250 300 350 400 450 500 549 599 649 699 749 799 849 899 949 999 Overdose Model Score
Machine Learning Approach Model Evaluation with the KS Statistic • KS was evaluated on all four test samples (25% holdout group) • During final testing, models were fit/tested against the full sample • Many commercial models have KS scores in the 35 to 50+ range Model KS Sample 1 KS Sample 2 KS Sample 3 KS Sample 4 Avg. KS Overdose Risk Score 75%/25% train/test 47.32 48.80 46.23 47.87 47.56 Overdose Risk Score 100%/100% train/test 48.34 49.62 47.57 48.27 48.45 Cumulative Percent of Population KS Plot from Sample 2 KS Score was 49.62
Recommend
More recommend
