Semantically Constrained Multilayer Annotation The Case of Coreference Jakob Prange, Nathan Schneider, and Omri Abend
How to annotate coreference? DMR, August 1, 2019 Prange, Schneider, Abend 2
Did anyone else have these fears ? DMR, August 1, 2019 Prange, Schneider, Abend 3
Did anyone else have these fears ? DMR, August 1, 2019 Prange, Schneider, Abend 4
Did anyone else have these fears ? DMR, August 1, 2019 Prange, Schneider, Abend 5
Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend 6
Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend 7
Representing Coreference Is Not Trivial What to annotate How to annotate How to annotate as mentions? mention spans? coreference? Syntactic criteria Minimum spans Identity, Bridging, … (e.g., all nouns) (only head words) Semantic criteria Maximum spans Unordered clusters (e.g., all events) (plus args & mods) or ordered chains Tricky linguistic Singletons? Hybrid? phenomena (e.g., coordination) DMR, August 1, 2019 Prange, Schneider, Abend 8
Representing Coreference Is Not Trivial • Many of these decisions seem arbitrary • Many different approaches with different guidelines [Poesio et al., 2016] • Problematic for evaluation [Moosavi and Strube, 2016; Moosavi et al., 2019] • Seldom integrated with other layers of meaning DMR, August 1, 2019 Prange, Schneider, Abend 9
DMR, August 1, 2019 Prange, Schneider, Abend H10
Our Approach: Build upon a basic framework for semantic units that can be shared among many higher-level meaning representations. DMR, August 1, 2019 Prange, Schneider, Abend 10
Our Approach: Build upon a basic framework for semantic units that can be shared among many higher-level meaning representations. DMR, August 1, 2019 Prange, Schneider, Abend H11
Our Approach: Build upon a basic framework for semantic units that can be shared among many higher-level meaning representations. DMR, August 1, 2019 Prange, Schneider, Abend 11
Semantic Multilayering UCCA DMR, August 1, 2019 Prange, Schneider, Abend H12
Semantic Multilayering Coreference UCCA DMR, August 1, 2019 Prange, Schneider, Abend H12
Semantic Multilayering World Knowledge Coreference UCCA DMR, August 1, 2019 Prange, Schneider, Abend H12
Semantic Multilayering World Knowledge Coreference UCCA DMR, August 1, 2019 Prange, Schneider, Abend H13
Semantic Multilayering World Knowledge Coreference UCCA DMR, August 1, 2019 Prange, Schneider, Abend H13
Semantic Multilayering with UCCA [Abend & Rappoport, 2013] • Identify and relate ”scenes” (events) and participants → Basic semantic units • No assumptions about grammar or lexicon → Cross-linguistically applicable DMR, August 1, 2019 Prange, Schneider, Abend 14
UCoref: UCCA + Coreference Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend 15
UCoref: UCCA + Coreference Scene-evoker Scene-evoker Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H16
UCoref: UCCA + Coreference Scene-evoker Scene-evoker Participant Participant Participant Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H16
UCoref: UCCA + Coreference Scene-evoker Scene-evoker Participant Participant Participant Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H16
UCoref: Streamlined Annotation • UCCA units can be filtered in preprocessing • All scene and participant units are automatically considered mentions DMR, August 1, 2019 Prange, Schneider, Abend 17
UCoref: Streamlined Annotation Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H18
UCoref: Streamlined Annotation Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H18
UCoref: Streamlined Annotation • UCCA units can be filtered in preprocessing • Human annotators identify remaining mentions and coreference clusters • Semantic heads serve as minimum span versions of complex mentions DMR, August 1, 2019 Prange, Schneider, Abend 19
UCoref: Streamlined Annotation Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H20
UCoref: Streamlined Annotation Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H20
UCoref: Streamlined Annotation Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H20
Comprehensive Representation ✓ Flexible mention spans ✓ Event and entity coreference in one framework ✓ Anchored in predicate argument structure, which… … helps humans disambiguate … can be utilized in automatic resolution (as features, or in a joint or MTL setup) DMR, August 1, 2019 Prange, Schneider, Abend 21
Drawback: Need UCCA Annotations First • Efficiently annotatable by non-experts • Can use automatic parsers • It’s worth it! DMR, August 1, 2019 Prange, Schneider, Abend 22
Pilot Annotation • Small samples from 3 English coreference corpora • OntoNotes [Hovy et al., 2006] : Blog posts • GUM [Zeldes, 2017] : WikiHow articles • RED [O’Gorman et al., 2016] : Forum discussions • Similar genres, different guidelines Hypothesis: UCoref covers (most of) what existing schemes cover, and more. DMR, August 1, 2019 Prange, Schneider, Abend 23
Hypothesis: UCoref covers (most of) what existing schemes cover, and more. OntoNotes GUM RED UCoref Anchored in syntax syntax tokens semantics Mention criteria syntax syntax semantics semantics Mention spans max max min flexible ✗ ( ✓ ) ✓ ✓ Events ✗ ( ✓ ) ✓ ✓ Singletons Annotation guidelines point in this direction. DMR, August 1, 2019 Prange, Schneider, Abend 24
Hypothesis: UCoref covers (most of) what existing schemes cover, and more. UCoref UCoref mentions referents mentions referents < < OntoNotes 40 128 20 96 GUM 288 466 155 291 Numbers of mentions and referents confirm: UCoref covers more than other schemes. DMR, August 1, 2019 Prange, Schneider, Abend 25
Hypothesis: UCoref covers (most of) what existing schemes cover, and more. UCoref UCoref mentions referents mentions referents ≈ ≈ RED 120 117 82 78 RED is very similar to UCoref in terms of coverage. DMR, August 1, 2019 Prange, Schneider, Abend 26
High Recall Exact Match 100 75 50 % 25 0 OntoNotes GUM RED Menmons Referents DMR, August 1, 2019 Prange, Schneider, Abend H27
Iterative greedy 1-to-1 alignment High Recall based on Dice coefficient. Exact Match Parmal Match 100 100 75 75 50 50 % 25 25 0 0 OntoNotes GUM RED OntoNotes GUM RED Menmons Referents DMR, August 1, 2019 Prange, Schneider, Abend H27
Iterative greedy 1-to-1 alignment High Recall based on Dice coefficient. Exact Match Parmal Match Mention spans are 100 100 crucial for 75 75 evaluation. 50 50 % 25 25 0 0 OntoNotes GUM RED OntoNotes GUM RED Menmons Referents DMR, August 1, 2019 Prange, Schneider, Abend H27
Iterative greedy 1-to-1 alignment High Recall based on Dice coefficient. Exact Match Parmal Match Mention spans are 100 100 crucial for Moosavi et al. [2019] 75 75 automatically extract evaluation. min spans. 50 50 % 25 25 0 0 OntoNotes GUM RED OntoNotes GUM RED Menmons Referents DMR, August 1, 2019 Prange, Schneider, Abend H27
Conclusions • Scattered coreference research needs to be unified • Related: Universal Coreference Initiative • Semantic representations should be semantically anchored and modular • UCoref is a first step in that direction • Main advantages: efficiency, consistency, and reusability DMR, August 1, 2019 Prange, Schneider, Abend 28
Ap Approa oaches es To Semantic Multilayer ering Highly modular Prague OntoNotes GUM [Böhmová et al., 2003] [Hovy et al., 2006] [Zeldes, 2017] Modular Decompositional Multi-sentence AMR Semantics [O’Gorman et al., 2018] [White et al., 2016] PropBank 1 Layer [Palmer et al., 2005] AMR FrameNet [Banarescu et al., 2013] RED [Baker et al., 1998] [O’Gorman et al., 2016] Sentence Semantics Syntax Token ANCHORING DMR, August 1, 2019 Prange, Schneider, Abend 29
Data & Code Paper Thank you! Questions? Jakob Prange, Nathan Schneider, and Omri Abend jakob@cs.georgetown.edu https://arxiv.org/abs/1906.00663 https://github.com/jakpra/UCoref
Appendix DMR, August 1, 2019 Prange, Schneider, Abend 31
Representing Coreference A: Did anyone else have these fears ? A: How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend 32
Recommend
More recommend
