Syntax for Semantic Role Labeling, To Be, Or Not To Be Shexia He 1,2 - PowerPoint PPT Presentation

Syntax for Semantic Role Labeling, To Be, Or Not To Be Shexia He 1,2 , Zuchao Li 1,2 , Hai Zhao 1,2,* , Hongxiao Bai 1,2 1 Department of Computer Science and Engineering, Shanghai Jiao Tong University 2 Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, China 1

Semantic Role Labeling (SRL) SRL - a shallow semantic parsing task: recognize the predicate-argument  structure, such as who did what to whom , where and when , etc. Four subtasks    Predicate identification and disambiguation   Argument identification and classification Applications:    Machine Translation   Information Extraction   Question Answering, etc. 2

SRL - Example Two formulizations of predicate-argument structure: Span-based (i.e., phrase or constituent)  Marry borrowed a book from john last week borrow.01 A0 A1 A2 AM-TMP Dependency-based: head of arguments  Marry borrowed a book from john last week borrow.01 A0 A1 A2 AM-TMP 3

Related Work  Previous methods Traditional Neural network Zhou and Xu (2015) introduced deep bi- directional RNN model Pradhan et al. (2005) utilized a SVM classifier Roth and Lapata (2016) proposed PathLSTM Roth and Yih (2005) employed CRF with modeling approach integer linear programming He et al. (2017) used deep highway BiLSTM Punyakanok et al. (2008) enforced global with constrained decoding consistency with ILP Marcheggiani et al. (2017) presented a Zhao et al. (2009) proposed a huge feature simple BiLSTM model engineering method Marcheggiani and Titov (2017) proposed a GCN-based SRL model 4

Focus - Dependency SRL Syntax-aware:   Maximum entropy model (Zhao et al., 2009)  Path embedding (Roth and Lapata, 2016)  Graph convolutional network (Marcheggiani and Titov, 2017) Syntax-agnostic:   The simple BiLSTM (Marcheggiani et al., 2017) 5

Method - Overview Pipeline   Predicate Disambiguation & Argument Labeling  Sequence labeling: BiLSTM - MLP  Enhanced representation: ELMo  Argument Labeling Model  Preprocessing: k -order pruning 6

k -order argument pruning  Initialization : Set the marked predicate as the current node;  1. Collect all its descendant node as argument candidates, which is at most k syntactically distant from the current node.  2. Reset the current node to its syntactic head and repeat step 1 until the root is reached.  3. Collect the root and stop. Reference: Zhao et al., 2009 7

syntax-aware syntax-agnostic CoNLL-2009 English training set CoNLL-2009 English development set 8

CoNLL-2009 Results Models English Chinese OOD Zhao et al., 2009 86.2 77.7 74.6 Non-NN Bjorkelund et al., 2010 85.8 78.6 73.9 Lei et al., 2015 86.6 - 75.6 FitzGerald et al., 2015 86.7 - 75.2 NN Roth and Lapata, 2016 86.7 79.4 75.3 syntax-aware Marcheggiani and Titov, 2017 88.0 82.5 77.2 Ours 89.5 82.8 79.3 Marcheggiani et al., 2017 87.7 81.2 77.7 NN syntax-agnostic Ours 88.7 81.8 78.8 Results on CoNLL-2009 English, Chinese and out-of-domain (OOD) test set. 9

End-to-end SRL  Integrate predicate disambiguation and argument labeling  CoNLL-2009 results Models F1 end-to-end 88.4 syntax-agnostic pipeline 88.7 end-to-end 89.0 syntax-aware pipeline 89.5 Results of end-to-end model on the CoNLL-2009 data. 10

CoNLL-2008 Results  Indispensable task: predicate identification Models LAS Sem-F1 Johansson and Nugues, 2008 90.13 81.75 Zhao and Kit, 2008 87.52 77.67 Zhao et al, 2009 88.39 82.1 89.28 82.5 Zhao et al, 2013 88.39 82.5 89.28 82.4 Ours (syntax-agnostic) - 82.9 Ours (syntax-aware) 86.0 83.3 Results on the CoNLL-2008 in-domain test set. 11

Syntactic Role Different syntax-aware SRL models may adopt different syntactic parser   PathLSTM SRL (Roth and Lapata, 2016): mate-tools  GCN-based SRL (Marcheggiani and Titov, 2017): BIST Parser How to quantitatively evaluate the syntactic contribution to SRL?   Evaluation Measure: the Sem-F 1 / LAS ratio  Sem-F 1 : the labeled F 1 score for semantic dependencies  LAS: the labeled attachment score for syntactic dependencies Reference: Surdeanu et al., CoNLL-2008 Shared Task 12

Performance Comparison Models LAS Sem-F1 Sem-F1/LAS Zhao et al, 2009 [CoNLL SRL-only] 86.0 85.4 99.3 Zhao et al, 2009 [CoNLL Joint] 89.2 86.2 96.6 Bjorkelund et al, 2010 89.8 85.8 95.6 Lei et al, 2015 90.4 86.6 95.8 Roth and Lapata, 2016 89.8 86.7 96.5 Marcheggiani and Titov, 2017 90.3 88.0 97.5 Ours + CoNLL-2009 predicted 86.0 89.5 104.0 Ours + Auto syntax 90.0 89.9 99.9 Ours + Gold syntax 100.0 90.3 90.3 Sem-F1/LAS ratio on CoNLL-2009 English test set. 13

Faulty Syntactic Tree Generator  How to obtain syntactic input of different quality?  A Faulty Syntactic Tree Generator (STG)  Produce random errors in the output parse tree  STG implementation  Given an input error probability distribution  Modify the syntactic heads of nodes 14

Sem-F1 - LAS Curve  Syntactic inputs generated from STG  The 10th-order SRL gives quite stable results regardless of syntactic quality  The 1st-order SRL model yields overall lower performance  Better syntax could result in better SRL 1st and 10th-order SRL on CoNLL-2009 English test set. 15

Conclusion and Future Work  We present an effective model for dependency SRL with extended k -order pruning.  The gap between syntax-enhanced and -agnostic SRL has been greatly reduced, from as high as 10% to only 1-2% performance loss.  High-quality syntactic parses indeed enhance SRL.  Future work:  Develop a more effective syntax-agnostic SRL system.  Explore syntactic integration method based on high-quality syntax. 16

Thank You! {heshexia, charlee}@sjtu.edu.cn Code is publicly available at: https://github.com/bcmi220/srl_syn_pruning