Oral presentation at MIE 2011: Factuality Levels of Diagnoses in - PowerPoint PPT Presentation

Oral presentation at MIE 2011: Factuality Levels of Diagnoses in Swedish Clinical Text Sumithra VELUPILLAI, Fil. Lic. (presenter) Hercules DALIANIS, Ph.D. Maria KVIST, M. D., PhD Oslo, August 30 th 2011 IT for Health, Dept. of Computer and Systems Sciences (DSV), SU, Dept. of clinical immunology and transfusion medicine, Karolinska University Hospital

Motivation • Improve information access – Reasoning documented in (EPR) free-text • speculations, negations, affirmations • important to distinguish – Accurate and situation-specific information – Overviews/summaries: these diagnoses have been affirmed, negated, … – Capture reasoning à deepened knowledge MIE 2011 Sumithra Velupillai 2

Aim and Objectives • Build automated information access systems – create an annotation schema for modeling factuality levels – applying this on Swedish clinical documentation – gain empirical understanding for qualitative analysis and use for automatic classifiers MIE 2011 Sumithra Velupillai 3

Methods • Creation of diagnosis list (~300 diagnoses) – Kvarst susp <Diagnosis>borrelia</Diagnosis> Remains (abbr.) suspected (abbr) borrelia • Annotation guidelines and annotations – Two senior physicians (“domain experts”) • Emergency ward, assessment entries – Stockholm EPR Corpus MIE 2011 Sumithra Velupillai 4

-­‑ ¡ + ¡ Certainly ¡ ¡ Probably ¡ ¡ Possibly ¡ Possibly ¡ Probably ¡ Certainly ¡ ¡ Posi.ve ¡ Posi.ve ¡ Posi.ve ¡ Nega.ve ¡ Nega.ve ¡ Nega.ve ¡ ¡ ¡ ¡ ¡ ¡ ¡ Patient has Parkinsons disease. Physical examination strongly suggests Parkinson. Patient possibly has Parkinson. Parkinson cannot yet be outruled. No support for Parkinson. Parkinsson can be excluded. MIE 2011 Sumithra Velupillai 5

Results • Intra- and Inter-Annotator Agreement – 0.7/0.58 F-measure, 0.73/0.6 Cohen’s κ • Certainly Positive clear majority (approx. 50%) – High IAA: 0.9 F-measure • Possibly Negative very rare • Discrepancies in intermediate classes MIE 2011 Sumithra Velupillai 6

Certainly ¡ ¡ Probably ¡ ¡ Possibly ¡ Possibly ¡ Probably ¡ Certainly ¡ ¡ Posi.ve ¡ Posi.ve ¡ Posi.ve ¡ Nega.ve ¡ Nega.ve ¡ Nega.ve ¡ 100% ¡ ¡ ¡ ¡ ¡ ¡ ¡ 90% ¡ 80% ¡ 70% ¡ 60% ¡ 50% ¡ 40% ¡ 30% ¡ 20% ¡ 10% ¡ 0% ¡ Annota.on ¡classes ¡for ¡15 ¡diagnoses ¡ MIE 2011 Sumithra Velupillai 7

Conclusion and discussion • Functional and agreeable model for annotators • IAA results suggest that this model can be used for developing automated systems • Different types of “cues” (not only linguistic) MIE 2011 Sumithra Velupillai 8

Where to go next? • Automatic classification – Keep model or collapse classes? – Classification methods? • Preliminary results promising (CRF) • Qualitative analysis – Subjective differences – Cues: linguistic, domain-specific, more? MIE 2011 Sumithra Velupillai 9

Thank you for your attention • Some key references V. Vincze, G. Szarvas, R. Farkas, G. Móra, J. Csirik, The Bioscope Corpus: Biomedical Texts Annotated for Uncertainty, Negation and their Scopes BMC Bioinformatics 9(S-11) (2008) R. Saurí, J. Pustejovsky FactBank: a corpus annotated with event factuality Language Resources & Evaluation 43 (2009), 227–268 Ideas and comments welcome! sumithra@dsv.su.se MIE 2011 Sumithra Velupillai 10