An Incremental Parser for Abstract Meaning Representation 1 Marco - PowerPoint PPT Presentation

An Incremental Parser for Abstract Meaning Representation 1 Marco Damonte 1 Shay B. Cohen 2 Giorgio Satta 1 University of Edinburgh 2 University of Padua EACL 2017 1 / 25

AMR He described her as a genius describe-01 ARG2 ARG1 ARG0 genius he she 2 / 25

Dependency trees • Transition-based dependency parsing (Nivre, 2004) root nsubj xcomp poss advmod My dog also likes eating 3 / 25

Concept identification The proposal 10 January 1989 The teacher person thing date-entity year day ARG0-of ARG1-of month 10 1989 propose-01 teach-01 1 4 / 25

Reentrancy I beg you to excuse me beg-01 ARG2 ARG1 ARG0 you excuse-01 ARG0 i ARG1 5 / 25

Reentrancy I beg you to excuse me beg-01 ARG2 ARG1 ARG0 you excuse-01 ARG0 i ARG1 6 / 25

Reentrancy I beg you to excuse me beg-01 ARG2 ARG1 ARG0 you excuse-01 ARG0 i ARG1 7 / 25

Transition-based AMR Parser 8 / 25

Transition system The boy wants to believe the girl STACK GRAPH boy 9 / 25

Transition system The boy wants to believe the girl STACK GRAPH want-01 want-01 boy boy 10 / 25

Transition system The boy wants to believe the girl STACK GRAPH want-01 want-01 boy 11 / 25

Transition system The boy wants to believe the girl STACK GRAPH want-01 believe-01 boy want-01 believe-01 12 / 25

Transition system The boy wants to believe the girl STACK GRAPH want-01 girl boy believe-01 believe-01 want-01 girl 13 / 25

Transition system The boy wants to believe the girl STACK GRAPH want-01 believe-01 boy want-01 believe-01 girl 14 / 25

Transition system The boy wants to believe the girl STACK GRAPH want-01 boy believe-01 believe-01 want-01 girl 15 / 25

Transition system The boy wants to believe the girl STACK GRAPH want-01 want-01 boy believe-01 girl 16 / 25

Transition system The boy wants to believe the girl STACK GRAPH want-01 boy believe-01 girl 17 / 25

Oracle • Given the current configuration ( σ, β, A ) and the gold-standard graph G = ( V g , A g ) :  LARC ( ℓ )    RARC ( ℓ )     T ( G , σ, β, A ) = RED-REENT ( ℓ )  REDUCE      SHIFT  • (English, AMR) ⇒ Transitions to obtain AMR* from English 18 / 25

Evaluation 19 / 25

Fine-grained evaluation • Smatch. Cai and Knight (2013) • Unlabeled. Smatch score after removing edge labels • No WSD. Smatch score while ignoring Propbank senses • Reentrancy. Smatch computed on reentrant edges • Semantic Role Labelling. Smatch computed on :ARG roles 20 / 25

Fine-grained evaluation • Smatch. Cai and Knight (2013) • Unlabeled. Smatch score after removing edge labels • No WSD. Smatch score while ignoring Propbank senses • Reentrancy. Smatch computed on reentrant edges • Semantic Role Labelling. Smatch computed on :ARG roles 20 / 25

Fine-grained evaluation • Smatch. Cai and Knight (2013) • Unlabeled. Smatch score after removing edge labels • No WSD. Smatch score while ignoring Propbank senses • Reentrancy. Smatch computed on reentrant edges • Semantic Role Labelling. Smatch computed on :ARG roles 20 / 25

Fine-grained evaluation • Smatch. Cai and Knight (2013) • Unlabeled. Smatch score after removing edge labels • No WSD. Smatch score while ignoring Propbank senses • Reentrancy. Smatch computed on reentrant edges • Semantic Role Labelling. Smatch computed on :ARG roles 20 / 25

Fine-grained evaluation • Smatch. Cai and Knight (2013) • Unlabeled. Smatch score after removing edge labels • No WSD. Smatch score while ignoring Propbank senses • Reentrancy. Smatch computed on reentrant edges • Semantic Role Labelling. Smatch computed on :ARG roles 20 / 25

Fine-grained evaluation (cont’d) • Concepts. F-score on the concept identification task • Negations. F-score on :polarity roles • Named Entities. F-score on :name roles • Wikification. F-score on :wiki roles 21 / 25

Fine-grained evaluation (cont’d) • Concepts. F-score on the concept identification task • Negations. F-score on :polarity roles • Named Entities. F-score on :name roles • Wikification. F-score on :wiki roles 21 / 25

Fine-grained evaluation (cont’d) • Concepts. F-score on the concept identification task • Negations. F-score on :polarity roles • Named Entities. F-score on :name roles • Wikification. F-score on :wiki roles 21 / 25

Fine-grained evaluation (cont’d) • Concepts. F-score on the concept identification task • Negations. F-score on :polarity roles • Named Entities. F-score on :name roles • Wikification. F-score on :wiki roles 21 / 25

Experiments Metric JAMR (’14) CAMR JAMR (’16) Ours Smatch 58 63 64 67 Unlabeled 61 69 69 69 No WSD 58 64 65 68 NP-only 47 54 55 58 Reentrancy 38 41 41 42 Concepts 79 80 83 83 Named Ent. 75 75 79 83 Wikification 0 0 64 75 Negations 16 18 45 48 SRL 55 60 60 56 JAMR: Flanigan et al. (2014) CAMR: Wang et al. (2015) 22 / 25

Software • Online demo: http://cohort.inf.ed.ac.uk/amreager.html • Source code for parser: https://github.com/mdtux89/amr-eager • Source code for evaluation: https://github.com/mdtux89/amr-evaluation 23 / 25

Demo 24 / 25

Conclusions • AMREager is a linear-time, left-to-right transition system • AMR parsing akin to dependency parsing • Fine-grained evaluation suite to assess AMR parsers 25 / 25