Validity & EHR-based Clinical Trials Bradley G Hammill Duke School of Medicine & Duke Clinical Research Institute brad.hammill@duke.edu
The Plan Introduce ADAPTABLE trial & PCORnet data Discuss process-based threats to validity Discuss data-based threats to validity
ADAPTABLE trial A spirin D osing: A P atient-centric T rial A ssessing B enefits and L ong-Term E ffectiveness – Pragmatic clinical trial – Demonstration project of PCORnet – Patient-level randomization – Leveraging EHR data “…designed to reflect ‘real - world’ – Events of interest primarily hospitalization-based medical care by recruiting broad populations of patients, embedding the – 20+ sites trial into the usual healthcare setting, and leveraging data from health systems to produce results that can be readily used to improve patient care.”
National Patient-Centered Clinical Research Network (PCORnet) Distributed Research Network – 13 Clinical Data Research Networks (CDRNs) comprising 80+ sites – Primarily electronic health record data – Use of Common Data Model (CDM) – Control of data is local, not central – Queries are used to generate summary results for return
PCORnet Common Data Model (CDM)
PCORnet Common Data Model (CDM)
Limitations of EHR Data Gaps in data capture exist… Patient X – For certain types of events Actual – For out-of-system encounters – Apr 2017, Recruited by Duke into study – Jun 2017, hospitalized @ Duke …that can lead to immediate validity issues – Aug 2017, hospitalized @ UNC – True event rate – Jan 2018, dies – Powered sample size Duke EHR – Site-level confounding – Jun 2017, hospitalized @ Duke
Addressing Limitations of EHR Data Addressing these gaps – Data linkage to outside sources – Pre-study gap analysis & selective site recruitment – Loyalty cohorts [observational studies]
Linkages within ADAPTABLE For ascertainment of events – Medicare claims data – Private health plan claims data (selected) – National Death Index – Direct records request
Medicare Claims Data Description – Medicare claims data reflect reimbursement for services requested by providers for beneficiaries enrolled in the traditional (fee-for-service) Medicare program Coverage – Subjects enrolled in fee-for-service Medicare (old age -or- disability) Known or anticipated limitations – Requires known & accurate linking information – Events ascertained using coding algorithms – Quarterly data is ~92% complete • Final / complete CY data is further delayed – ~25% of Medicare population is enrolled in a managed care plan (i.e., no claims)
Private Health Plan Claims Data Description – PCORi has funded a demonstration project with Anthem and Humana to provide health plan data for ADAPTABLE subjects Coverage – Subjects enrolled in an Anthem or Humana health plan Known or anticipated limitations – Requires known & accurate linking information – Events ascertained using coding algorithms – Limited geographic coverage – Turnover within plans can be substantial
National Death Index Data Description – The National Death Index is a centralized database of death record information on file in state vital statistics offices Coverage – All subjects Known or anticipated limitations – Requires known & accurate linking information – Early release data is ~90% complete • Final / complete CY data is further delayed
Direct Records Request Description – Patient-reported events that cannot be reconciled using other sources will be followed up on by the ADAPTABLE call center Coverage – All subjects Known or anticipated limitations – Requires patient report to trigger reconciliation – Paper records will be returned & events adjudicated
Data Latency 𝑢 𝐸𝐵𝑈𝐵 : Time delay between latest available data and acquisition – Present for some sources – Differs by source – Possible reasons: Accrual time; request processing time Data available DSMB 𝑢 𝑄𝑆𝑃𝐷𝐹𝑇𝑇 : Time required for pre-processing data at coordinating center 𝑢 𝐸𝐵𝑈𝐵 𝑢 𝑄𝑆𝑃𝐷𝐹𝑇𝑇 𝑢 𝑇𝑈𝐵𝑈𝑇 – Present for some sources (esp. claims) – Assuming uniform for all sources when present (1 month) 𝑢 𝑇𝑈𝐵𝑈𝑇 : Time required for processing and analyzing data – Uniform for all sources
Data Latency PCORnet EHR data – 𝑢 𝐸𝐵𝑈𝐵 : Best case ~1 month; worst case ~7+ months • DataMarts refreshed every 6 months • Some source tables may not be up-to-date at time of refresh • Must be certified for use by PCORnet operations center – 𝑢 𝑄𝑆𝑃𝐷𝐹𝑇𝑇 required? No Data available DSMB Medicare claims data 𝑢 𝐸𝐵𝑈𝐵 𝑢 𝑄𝑆𝑃𝐷𝐹𝑇𝑇 𝑢 𝑇𝑈𝐵𝑈𝑇 – 𝑢 𝐸𝐵𝑈𝐵 : Best case ~6 months; worst case ~9 months • Quarterly data available ~5 months following the end of the quarter • Acquisition time required (~1 month) – 𝑢 𝑄𝑆𝑃𝐷𝐹𝑇𝑇 required? Yes (~1 month)
Data Latency Private health plan claim data – 𝑢 𝐸𝐵𝑈𝐵 ~4 months • ~3-month lag in claims to insurer • Acquisition time required (~1 month) – 𝑢 𝑄𝑆𝑃𝐷𝐹𝑇𝑇 required? No Data available DSMB National Death Index data 𝑢 𝐸𝐵𝑈𝐵 𝑢 𝑄𝑆𝑃𝐷𝐹𝑇𝑇 𝑢 𝑇𝑈𝐵𝑈𝑇 – 𝑢 𝐸𝐵𝑈𝐵 : Best case ~2 months; worst case ~13+ months • Early release data available 2-3 months after the end of the CY • Acquisition time required (~1 month) – 𝑢 𝑄𝑆𝑃𝐷𝐹𝑇𝑇 required? Yes (~1 month) Assuming 𝑢 𝑇𝑈𝐵𝑈𝑇 = 1 month for all data sources
Impact of Data Latency Data Through… Data Source Mytrus Patient Portal 1-Nov-2017 PCORnet DataMart (recent refresh) 1-Oct-2017 PCORnet DataMart (distant refresh) 1-Apr-2017 Medicare Claims Data 30-Jun-2017 National Death Index 31-Dec-2016 Private Health Plan Data 1-Aug-2016 Apr-2016 Dec-2017 DSMB
Information Asynchrony Events found in EHR / Medicare / PHP data are accepted as true Patient-reported events must be reconciled – Search EHR? Found = Confirmed. Unfound… – Search Medicare? Found = Confirmed. Unfound… – Search Private Health Plan data? Found = Confirmed. Unfound… – Call for medical records Data latency affects timing of event recording (by trial) and reconciliation
Information Asymmetry By type of data – Raw hospital records vs. coded hospital records Patient #1 EHR CMS NDI HP By sources of data Patient #2 EHR CMS NDI HP – Different patients can have different sources of data contributing to endpoint ascertainment Patient #3 EHR CMS NDI HP Patient #4 EHR CMS NDI HP
Potential Data Issues in PCORnet Validity of coded endpoints Quality of data at PCORnet sites Identifying appropriate patients for recruitment (computable phenotype)
Validity of Coded Endpoints ADAPTABLE events – Death – Hospitalization for non-fatal MI – Hospitalization for stroke – Coronary revascularization – Hospitalization for major bleeding Not exclusively an EHR issue, but… Definitions vastly different from “regular” trials
Definitions of Myocardial Infarction EHR criteria Adjudication criteria Inpatient encounter w/ ECG or changes consistent with acute infarction or ischemia MI: ICD-9-CM diagnosis code 410.x0, • New diagnostic Q waves (Q wave in leads V2 and V3 ≥ 0.02 sec or QS complex in leads 410.x1 in primary position V2 and V3; Q wave ≥ 0.03 sec and ≥ 0.1 mV deep or QS complex in leads I, II, aVL, aVF or V4-V6 in any two leads of a contiguous lead grouping (I and aVL; V1-V6; II, III, aVF, R wave ≥ 0.04 sec in V1 and V2 and R/S ≥ 1 with a concordant positive T wave)) in the absence of conduction abnormalities • New significant ST -segment-T -wave changes in two or more contiguous leads: ST elevation at the J point ≥ 0.1 mV in all leads other than leads V2 and V3 where the following cut points apply: ≥ 0.2 mV in men ≥ 40 years; 0.25 mV in men < 40 years, or ≥ 0.15 mV in women. ST depression horizontal or downsloping ≥ 0.05 mV; or T wave inversion ≥ 0,1mV with prominent R wave or R/S ratio ≥ 1. • Development of new left bundle branch block (LBBB) • Imaging evidence of new loss of viable myocardium or new regional wall motion abnormality Intracoronary thrombus by angiography • AND Elevated cardiac biomarkers (values according to each hospital’s laboratory): A rise and/or fall in cardiac biomarker values (preferably troponin, CKMB, AST, LDH or myoglobin) with at least one value above the 99th percentile of the upper reference limit.
ADAPTABLE Validation Studies EHR-coded events vs. adjudicated events Patient-reported events vs. coded events
Related PCORnet Hospitalization Issue Not all sites code “primary” diagnosis for hospitalizations – Effect = No events at a site? Address by… – Making this a required field for site inclusion – Defining alternative endpoints Myocardial infarction – Primary: Inpatient encounter w/ICD-9-CM diagnosis code 410.x0, 410.x1 in primary position – Alternate: Inpatient encounter w/ICD-9-CM diagnosis code 410.x0, 410.x1 in any position
Recommend
More recommend
