Radiology Reporting with Life: Three Years' Experience using Hyperlinked Interactive Multimedia Reporting Adoption and Value: Objective Evaluation via Click Through Rates Authors: Folio LR, Cohen G, Machado LB Radiology and Imaging Sciences, Clinical Center National Institutes of Health Presenting author: Les Folio, DO, MPH, MSc, MAS Col (ret) USAF Director, Clinical Image Processing Service and Lead Radiologist for CT, NIH CC Adjunct Clinical Professor of Radiology, George Washington University Hospital @LesFolio #SIIM18
Disclosures, Disclaimers, Conflicts of Interest Presenter manages a research agreement with Carestream Health (the NIH Clinical Center PACS shown in this presentation) Presenter has government issued diagnostic imaging patents (unrelated; no royalties) Presenter receives book author royalties (Springer) This research was supported [in part] by the NIH Clinical Center Intramural Research Program The content is the responsibility of the presenter and does not necessarily represent the official views of the National Institutes of Health @LesFolio t #SIIM18 s
Objectives • Graphically depict adoption rates of hyperlinks in interactive reporting • over time since we implemented in Feb 2015 • among radiology subspecialties and modalities supporting efficiency • Analyze clinicians/radiologists interaction with EMR, PACS, reports • and report hyperlinks evidenced by click through analysis by referring medical subspecialties • Objectively assess report value by exploring multimedia report interaction • by click through behaviors of referring clinicians and radiologists Work in progress, AI: link to annotation supervised deep learning • how interactive reports help mitigate the deep learning missing “link” @LesFolio t #SIIM18
Background • Radiology reports have not dramatically changed in over 100 years • Since Roentgen’s discovery (1895) • Radiologist and oncologists surveyed on reporting preferences • verified oncologists and radiologists prefer hyperlinks • Descriptions in our reports are hyperlinked to measurements • to the annotation, providing unique opportunities: • to objectively analyze referring physicians Click Through Rate (CTR) • provides valuable labeling for Artificial Intelligence (AI)/ deep learning * Folio L. Quantitative Radiology Reporting and Tumor Metrics: @LesFolio t #SIIM18 Survey of Oncologists and Radiologists. AJR. Oct 2015.
Background (cont.) Template-based reports used 100 years ago… Example of Dr. Carman’s report (1913), Mayo Clinic Russell D. Carman 1875-1926 Head of Roentgenology at the Mayo Clinic @LesFolio #SIIM18 Source: “Radiology: an illustrated history” by Ronald L. Eisenberg
Example Hyperlink Insertion while Dictating Record FINDINGS : Chest CT: Lungs, pleurae: Unchanged lung nodules for example right upper lobe (1.9 cm x 1.5 cm) (series 4, image 81) • Minimizes crosscheck • Metadata automatically includes: x,y,z location, who measured, when, relation and Series 4 designation, name, lesion type Image 81 • “Active annotation” is either most recent measured, clicked or “b” shortcut @LesFolio t #SIIM18
@LesFolio t #SIIM18
@LesFolio t #SIIM18
@LesFolio t #SIIM18
Multimedia Enhanced Radiology Reports FINDINGS : Chest CT: Lungs, pleurae: Unchanged lung nodules for example right upper lobe (0.8 cm x 0.4 cm) (series 4, image 84) Mediastinum, heart, great vessels: Unchanged mediastinal adenopathy for example subcarinal (2.5 cm x 1.4 cm) (series 2, image 27) and right hilar adenopathy for example (5.1 cm x 2.4 cm) (series 2, image 32) and (2.1 cm x 1.4 cm) (series 2, image 25) Abdomen CT: Lymph nodes, abdominopelvic vascular: unremarkable Liver, spleen, biliary, gallbladder, pancreas: unremarkable GU Kidneys, ureters, adrenal glands: unremarkable GI Small and large bowel, mesentery, peritoneum: unremarkable Pelvic CT: Central pelvis, sidewalls: Unchanged anterior pelvic wall mass. Osseous structures, spine, body wall, soft tissues: unremarkable IMPRESSION : 1. Unchanged lung nodules 2. Stable mediastinal and hilar adenopathy/masses 3. Unchanged anterior pelvic wall masses 4. No evidence of new soft tissue mass @LesFolio t #SIIM18
Mu Mult ltim imedia ia a and Hyperlin links F s Facili ilitate Repor ortin ing f for or Clin linic ical Tria rials ls RECIST 1.1 Radiology Report Impression: Evaluation of target lesions Disappearance of target lesions “Stable metastatic lesions” Complete Response (CR) (LN<1cm) ≥ 30% decrease from baseline sum of Partial Response (PR) Partial Response = Hope target lesions size ≥ 20% increase from baseline or best Stable Disease = “therapy is not reducing my cancer” response * + absolute increase ≥ 5mm Progressive Disease (PD) on target or Non target lesions / New Patients can get conflicting messages in Patient Portal lesions Stable Disease (SD) Neither CR or PD This number is what “counts” @LesFolio t #SIIM18
@LesFolio t #SIIM18
Creation of Bookmarks 1 & Report Hyperlinks 2 (by Modality) Hyperlinks 3 implemented Feb 2015 July 2014 – Dec 2017 1 Bookmarks = any image annotation Not all bookmarks are hyperlinked Bookmark capability preceded links 2 Hyperlinks = hyperlinked text-to-annotation Requires a bookmark 3 We had bookmark capability before hyperlinks @LesFolio t #SIIM18
Creation of Bookmarks & Report Hyperlinks (Modality & Subspecialty) July 2016 – Dec 2017 @LesFolio t #SIIM18
Adoption by Radiologists • Radiologists immediately implemented interactive reporting • Shortly after capability was implemented • Expected subspecialty and modality variation • Body radiologists adopted more than neuro radiologists • Also greater adoption for CT and PET-CT than MRI • NIH adoption was greater than other centers; perhaps because of NIH emphasis on oncology and clinical trials @LesFolio t #SIIM18
Click Through Analysis • Objective evaluation value of report hyperlinks to referring docs • Also to radiologists; we show body radiologist CTR • CTR is an indirect, yet objective measure of report value • A clinician must open a report to click; usually indicates reading them • We target PACS & EMR capability refreshers to infrequent users • To those clicking on report hyperlinks least often • Surgeons infrequent @LesFolio t #SIIM18
Body Radiologists click throughs of bookmarks and hyperlinks Access from PACS to report bookmarks and hyperlinks by BODY (CT+MR) Radiologists Dec 2017 More studies 1600 are accessed 1403 1400 # of studies 1116 than created 1200 1000 822 800 600 400 200 0 Total created* Total created with Total read and bookmarks or bookmark or hyperlinks** hyperlink clicked*** * Total CT and MR studies with reports signed by Body radiologists ** As above but where the study contains bookmarks or the report contains hyperlinks Body radiologists find value in bookmarks they create for follow up purposes *** Total CT and MR Body studies with bookmarks or hyperlinks where these were clicked at least once by a radiologist @LesFolio t #SIIM18
Physician Clicks from EMR to Thin Client Viewer Showing Images/Report About 80% of click throughs from our EMR to our PACS are from four NIH institutes. Example action: Can help target EMR/ PACS training Institute Report ”Click Through” analysis confirms value of imaging exams and reports @LesFolio t #SIIM18
Discussion • Revolutionary technologic implementation (hyperlinks) often challenging • widespread adoption support improved efficiency • Applying advanced hyperlink analytic tools to radiology reports • can be objective evidence of report value • Study limitations include a small sample and unique capability • based on our experience of interactive reporting • we are confident our sample is representative of our hyperlink use • annotations can be distracting (initial complaints) • training provided to show Ctrl+G hides graphics; also “show only mine” @LesFolio t #SIIM18
Optimal Radiologist Annotations? Intersection over union (IoU) and paired t-tests: Linear, two-diameter, and oval were 0.29±0.23, 0.70±0.22, and 0.73±0.15 respectively. @LesFolio t #SIIM18
Supervised Deep Learning Model of Annotated Lung CT images Interconnected with Hyperlinks in Interactive Radiology Reports Do M, Folio L, Machado L. APECED Deep Learning. SCBTMR 2018. @LesFolio t #SIIM18
DeepLesion: 32,735 measured lesions/ bounding boxes (soon to be publically available) @LesFolio t #SIIM18
Real estate mantra (Location 3 ) twist: Location, Labeling, Learning @LesFolio t #SIIM18
Conclusions • We demonstrated adoption of bookmark and hyperlink use • subdivided by radiology subspecialty; supporting efficiency • body radiologists had the highest hyperlink usage • We objectively assessed report value by referring clinicians & institutes • by analyzing CTR of hyperlinks within our EMR, reports • Report text lined directly to annotations help fill missing DL link Manuscript Submitted to JDI May 2018. @LesFolio t #SIIM18
Thank you Les.Folio@nih.com @LesFolio t #SIIM18
Recommend
More recommend
