Tree-Adjoining Grammar Parsing and Vector Representations of - PowerPoint PPT Presentation

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Tree-Adjoining Grammar Parsing and Vector Representations of Supertags Jungo Kasai Yale University December 14, 2017

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Outline Background and Motivations 1 Supertagging Models 2 Parsing Models 3 Vector Representations of Supertags 4 Ongoing TAG Parsing Work 5 Applications of TAG 6 Future Work 7

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Outline Background and Motivations 1 Supertagging Models 2 Parsing Models 3 Vector Representations of Supertags 4 Ongoing TAG Parsing Work 5 Applications of TAG 6 Future Work 7

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Syntactic Parsing S NP VP John AdvP VP really V NP likes Mary Why do we need parsing? Does John love Mary? Does Mary love John? Understanding of a sentence depends on the structure

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Context Free Grammars S S → NP VP VP → AdvP VP NP VP AdvP → really VP → V NP John AdvP VP NP → Mary NP → they really V NP NP → John V → like likes Mary V → likes These production rules generate sentences

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Context Free Grammars S S → NP VP VP → AdvP VP NP VP AdvP → really VP → V NP John AdvP VP NP → Mary NP → they really V NP NP → John V → like likes Mary V → likes Fundamental problem : constraints are distributed over separate rules How do we choose V → like or V → likes ?

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Tree-Adjoining Grammar Tree-Adjoining Grammar (TAG) localizes grammatical constraints Finite set of lexicalized elementary trees Finite set of operations (Substitution and Adjunction) are used to combine elementary trees S VP S VP * NP 0 ↓ VP AdvP NP 0 ↓ VP NP Ad ♦ V ♦ V ♦ NP 1 ↓ N ♦ sleep really likes John

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Tree-Adjoining Grammar Substitution

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Tree-Adjoining Grammar Adjunction

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Tree-Adjoining Grammar Adjunction allows for unbounded recursion while still enforcing agreement. John smartly occasionally really only likes Mary...

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Derivation Tree Derivation tree records the operations. Forms a dependency tree (each token has exactly one parent) likes Subst 0 AdjSubst 1 John really Mary ROOT Subst 0 ADJ Subst 1 ROOT John really likes Mary

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Two Steps in TAG Parsing Now the reverse process. Supertagging Assign elementary trees (supertags) to each token. Similar to POS tagging. Parsing Predict operations on the elementary trees. NP * NP * S S NP 0 ↓ S NP 0 ↓ VP NP VP V ♦ -NONE- V ♦ left left

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Outline Background and Motivations 1 Supertagging Models 2 Parsing Models 3 Vector Representations of Supertags 4 Ongoing TAG Parsing Work 5 Applications of TAG 6 Future Work 7

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Supertagging is a bottleneck Supertagger Parser Stag Acc UAS LAS Gold Chart (MICA) 100.00 97.60 97.30 Maxent (MICA) Chart (MICA) 88.52 87.60 85.80 Supertagging is almost parsing There are about 5,000 supertags in the grammar About half of them occur only once in the training data (PTB WSJ Sections 1-22).

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W BiLSTM Supertagging Figure: BiLSTM Supertagger Architecture.

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Supertagging is still a bottleneck Supertagger Parser Stag Acc UAS LAS Maxent (MICA) Chart (MICA) 88.52 BiLSTM Chart (MICA) 89.32

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Supertagging is still a bottleneck Supertagger Parser Stag Acc UAS LAS Gold Chart (MICA) 100.00 97.60 97.30 Maxent (MICA) Chart (MICA) 88.52 87.60 85.80 BiLSTM Chart (MICA) 89.32 90.05 88.32 We can compensate for supertagging errors by exploiting structural similarities across elementary trees. Similarities across supertags are not utilized by the chart parser. We use two alternative families of parsing algorithms

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Outline Background and Motivations 1 Supertagging Models 2 Parsing Models 3 Vector Representations of Supertags 4 Ongoing TAG Parsing Work 5 Applications of TAG 6 Future Work 7

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Parsing Models Prior Work: Unlexicalized Chart-Parser (MICA) [Bangalore et al., 2009] Unlexicalized Transition-based Parser [Kasai et al., 2017, Friedman et al., 2017] Graph-based Parser (work in progress)

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Transition-based Parsing Arc-Eager System (MALT) [Nivre et al., 2006] ROOT Subst 0 ADJ Subst1 ROOT John really likes Mary

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Transition-based Parsing

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Transition-based Parsing

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Transition-based Parsing

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Transition-based Parsing

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Transition-based Parsing

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Transition-based Parsing

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Transition-based Parsing

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Transition-based TAG Parsing How do we learn? Represent the configuration by the top k elements from stack and buffer: { s i , b i } k i = 1 [Chen and Manning, 2014]. Represent s i ( b i ) by the TAG elementary tree and the derived substitution operations performed into s i . Encode the TAG elementary trees and the substitution operations with dense vectors.

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W NN Transition-based Parsing Model Figure: Transition-based Parser Neural Network Architecture.

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Example John really likes Mary Stack Buffer Relations Action ROOT likes Mary { ( ROOT , likes , ROOT ) , ( likes , John , 0 ) · · · } RIGHT:1

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Parsing Results Gold Stags Predicted Stags Parsing Model UAS LAS UAS LAS MICA Chart 97.60 97.30 90.05 88.32 Transition-based 97.67 97.45 90.23 88.77 Table: Results on Section 00. Beam size 16. Predicted supertags are from our BiLSTM supertagger.

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Outline Background and Motivations 1 Supertagging Models 2 Parsing Models 3 Vector Representations of Supertags 4 Ongoing TAG Parsing Work 5 Applications of TAG 6 Future Work 7

Background and Motivations Supertagging Models Parsing Models Vector Representations of Supertags Ongoing TAG Parsing W Embeddings for Elementary Trees