External Validity of NYC Macroscope Electronic Health External Validity of NYC Macroscope Electronic Health External Validity of NYC Macroscope Electronic Health External Validity of NYC Macroscope Electronic Health Record Surveillance System Indicator Definitions of Obesity, Record Surveillance System Indicator Definitions of Obesity, Record Surveillance System Indicator Definitions of Obesity, Record Surveillance System Indicator Definitions of Obesity, Smoking, Diabetes, Hypertension and Hypercholesterolemia Smoking, Diabetes, Hypertension and Hypercholesterolemia Smoking, Diabetes, Hypertension and Hypercholesterolemia Smoking, Diabetes, Hypertension and Hypercholesterolemia Katharine H. McVeigh, PhD, MPH New York City Department of Health and Mental Hygiene Presented at Concordium September 13, 2016
NYC Macroscope Team NYC Macroscope Team NYC Macroscope Team NYC Macroscope Team NYC Department of Health and Mental Hygiene Katharine H. McVeigh, PhD, MPH Sharon E. Perlman, MPH Elizabeth Lurie, MPH Pui Ying Chan, MPH Kathleen Tatem, MPH Sungwoo Lim, DrPH Laura Jacobson, MSPH Lauren Schreibstein, MA City University of New York School of Public Health Lorna E. Thorpe, PhD Special thanks to Jay Bala, Katherine Bartley, Claudia Chernov, Amy Freeman, Ryan Grattan, Carolyn Greene, Tiffany Harris, Stephen Immerwahr, Kevin Konty, Ram Koppaka, Remle Newton-Dame, Jesica Rodriguez-Lopez, Matthew Romo, Sarah Shih, Jesse Singer, Elisabeth Snell This work has been made possible by the financial support of the de Beaumont Foundation, the Robert Wood Johnson Foundation and its National Coordinating Center for Public Health Services and Systems Research, the Robin Hood Foundation, the NY State Health Foundation, the Doris Duke Charitable Foundation, and the Centers for Disease Control and Prevention (U28EH000939)
Electronic Health Records (EHRs) Can Transform Electronic Health Records (EHRs) Can Transform Electronic Health Records (EHRs) Can Transform Electronic Health Records (EHRs) Can Transform Medical Records into Actionable Information Medical Records into Actionable Information Medical Records into Actionable Information Medical Records into Actionable Information 80 70 60 50 40 30 20 Oct-09 Dec-09 Feb-10 Apr-10 Jun-10 Aug-10 Oct-10 Dec-10 Feb-11 Apr-11 Jun-11 Aug-11 Oct-11
Why is Electronic Health Record Why is Electronic Health Record Why is Electronic Health Record Why is Electronic Health Record- - - -Based Based Based Based Surveillance Important? Surveillance Important? Surveillance Important? Surveillance Important? • The burden of chronic disease in the U.S. is increasing. • Population-level monitoring of disease and risk factor prevalence is important for prevention and mitigation • Traditional surveys are becoming more difficult to carry out • Telephone survey response rates are dropping • Examination surveys are extremely expensive, labor intensive, and often have lengthy lag times between data collection and dissemination
What is the NYC Macroscope? What is the NYC Macroscope? What is the NYC Macroscope? What is the NYC Macroscope? • An electronic health record-based surveillance system for New York City, focusing on chronic disease and risk factors • Developed by the New York City Department of Health and Mental Hygiene in collaboration with the City University of New York School of Public Health
Key Features of NYC Macroscope (1) Key Features of NYC Macroscope (1) Key Features of NYC Macroscope (1) Key Features of NYC Macroscope (1) • Based on a distributed data model • Hub Population Health System • eClinicalWorks EHR platform • Inclusion/exclusion criteria • Practice – Documentation quality thresholds • Provider – Primary care only • Patient – Visit in 2013, ages 20-100, sex recorded as male or female, NYC Zip Code • Record – Lab measures require electronic reporting, records with missing data are dropped before weighting
Key Features of NYC Macroscope (2) Key Features of NYC Macroscope (2) Key Features of NYC Macroscope (2) Key Features of NYC Macroscope (2) • Weighted to the distribution of the NYC adult population that had seen a health provider in the past year • N = 716,076 patients seen in 2013 ( 17.5% of 4.1 M in care ) • Age group (20-39, 40-59, 60-100) • Sex (male, female) • Neighborhood poverty level (< 10%, 10-19%, 20-29%, >= 30%) • Validated against 2 population-based reference surveys • 2013-14 NYC Health and Nutrition Examination Survey (NYC HANES) • N = 1,524; 1,135 in care • 2013 NYC Community Health Survey (CHS) • N* = 8,356; 6,166 in care
Population Estimate Comparisons Population Estimate Comparisons Population Estimate Comparisons Population Estimate Comparisons NYC Macroscope, NYC HANES and CHS Prevalence Estimates (%) 0 10 20 30 40 50 60 Obesity Smoking Diabetes Diagnosis *Augmented Diabetes Hypertension Diagnosis *Augmenented Hypertension Hypercholesterolemia Diagnosis *Augmented Hypercholesterolemia NYC Macroscope NYC HANES CHS * Not available in CHS
Medical Chart Review Medical Chart Review Medical Chart Review Medical Chart Review Criterion Criterion Criterion Criterion- -Related - - Related Related Validity (N = 48) Related Validity (N = 48) Validity (N = 48) Validity (N = 48) Sensitivity Specificity Obesity Obesity Smoking Smoking Diabetes Diagnosis Diabetes Diagnosis Augmented Diabetes Augmented Diabetes Hypertension Diagnosis Hypertension Diagnosis Augmenented Hypertension Augmenented Hypertension Hypercholesterolemia Hypercholesterolemia Diagnosis Diagnosis Augmented Augmented Hypercholesterolemia Hypercholesterolemia 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
External Validity Study External Validity Study External Validity Study External Validity Study Is the validity of NYC Macroscope indicators generalizable to EHR data maintained by other providers on other platforms?
Research Questions Research Questions Research Questions Research Questions • What is the criterion-related validity of NYC Macroscope indicator definitions in data from practices that do not contribute to the NYC Macroscope? • Does validity improve if records are restricted to providers who have attested to stage 1 meaningful use?
Participant Inclusion/Exclusion Flow Participant Inclusion/Exclusion Flow Participant Inclusion/Exclusion Flow Participant Inclusion/Exclusion Flow Chart Chart Chart Chart Enrolled in NYC HANES 2013-14 N=1,524 Had a doctor visit in past year Not in care n=1,135 n=389 Signed consent No consent n=692 n=443 Signed HIPAA waiver No HIPAA waiver n=491 n=201 No EHR, no visits, specialist, One or more EHRs obtained unable to locate, not released n=214 n=277 Excluded provider type EHR contained valid data n=87 n=190 NYC Macroscope records Non-Macroscope records n=48 n=142
Methods Methods Methods Methods • 142 non-Macroscope records • 133 providers • 89 medical practices • > 20 different EHR vendor platforms • Sensitivity and Specificity • Full sample • Sub-sample of 86 records (MU1)
Sample Sample Sample Characteristics Sample Characteristics Characteristics Characteristics Non-Macroscope Records NYC MU1 Macroscope All Records Subsample (n=48) (n=142) (n=86) % % % Age Group 20-39 35 36 35 40-49 46 37 34 ≥ 60 19 28 31 Sex Female 65 65 70 Male 35 35 30 Neighborhood Poverty < 10% 27 23 21 10%-19% 33 36 34 20%-29% 21 25 31 ≥ 30% 19 16 14
Sensitivity Sensitivity Sensitivity Sensitivity Obesity Smoking Diabetes Diagnosis Augmented Diabetes Hypertension Diagnosis Augmenented Hypertension Hypercholesterolemia Diagnosis Augmented Hypercholesterolemia 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 All Non-Macroscope Records (n=142) Non-Macroscope with MU1 Restriction (n=86) NYC Macroscope (n=48) Validity threshold ≥ 0.70
Specificity Specificity Specificity Specificity Obesity Smoking Diabetes Diagnosis Augmented Diabetes Hypertension Diagnosis Augmenented Hypertension Hypercholesterolemia Diagnosis Augmented Hypercholesterolemia 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 All Non-Macroscope Records (n=142) Non-Macroscope with MU1 Restriction (n=86) NYC Macroscope (n=48) Validity threshold ≥ 0.80
Summary Summary Summary Summary • Both indicators of hypercholesterolemia performed poorly • All other measures performed well • The above conclusions are consistent with findings from the 48 NYC Macroscope records • Restricting records to those from providers who have attested to stage 1 meaningful use meaningfully improved the sensitivity of the smoking and hypertension diagnosis indicators
Strengths and Limitations Strengths and Limitations Strengths and Limitations Strengths and Limitations • Strengths • Heterogeneity of providers (N = 133) and EHR vendor platforms (N > 20) • Innovative sample and gold standard criterion • Limitations • Small sample size/large confidence intervals
Recommend
More recommend
