Global Neural CCG Parsing with Optimality Guarantees Kenton Lee Mike Lewis † Luke Zettlemoyer University of Washington UWNLP † Now at Facebook AI Research 1
This Talk Challenge : Global models (e.g. Recursive NNs) break dynamic programs S NP S \ NP NP/NP NP ( S \ NP ) /NP NP Fruit flies like bananas 2
This Talk Challenge : Global models (e.g. Recursive NNs) break dynamic programs S NP S \ NP Our approach : Combine local and global models in NP/NP NP ( S \ NP ) /NP NP A * parser Fruit flies like bananas Result : Global model with exact inference 3
Parsing with Hypergraphs Fruit flies like bananas Input Output Fruit flies like bananas NP/NP NP ( S \ NP ) /NP NP > > NP S \ NP < S 4 Klein and Manning, 2001
Parsing with Hypergraphs Fruit flies like bananas Input Output Fruit flies like bananas NP/NP NP ( S \ NP ) /NP NP > > NP S \ NP < S Fruit flies like bananas NP/NP NP ( S \ NP ) /NP NP like bananas Fruit flies ( S \ NP ) /NP NP NP/NP NP > > > NP S \ NP > NP S \ NP < S Fruit like flies bananas NP/NP NP ( S \ NP ) /NP NP ∅ 5 Klein and Manning, 2001
Parsing with Hypergraphs Fruit flies like bananas Input Output Fruit flies like bananas NP/NP NP ( S \ NP ) /NP NP > > NP S \ NP < S Fruit flies like bananas NP/NP NP ( S \ NP ) /NP NP like bananas Fruit flies ( S \ NP ) /NP NP NP/NP NP > > > NP S \ NP > NP S \ NP < S Nodes represent Fruit like flies bananas NP/NP NP ( S \ NP ) /NP NP partial parses ∅ 6 Klein and Manning, 2001
Parsing with Hypergraphs Fruit flies like bananas Input Output Fruit flies like bananas NP/NP NP ( S \ NP ) /NP NP > > NP S \ NP < S Fruit flies like bananas NP/NP NP ( S \ NP ) /NP NP like bananas Fruit flies ( S \ NP ) /NP NP NP/NP NP > > > NP S \ NP > NP S \ NP < S Fruit like flies bananas Hyperedges represent NP/NP NP ( S \ NP ) /NP NP rule productions ∅ 7 Klein and Manning, 2001
Parsing with Hypergraphs Fruit flies like bananas Input Output Fruit flies like bananas NP/NP NP ( S \ NP ) /NP NP > > NP S \ NP < S Fruit flies like bananas NP/NP NP ( S \ NP ) /NP NP like bananas Fruit flies ( S \ NP ) /NP NP NP/NP NP > > > NP S \ NP > NP S \ NP < S Fruit like flies bananas Path y = { e 1 , . . . , e m } NP/NP NP ( S \ NP ) /NP NP represents a parse derivation ∅ 8 Klein and Manning, 2001
Parsing with Hypergraphs Fruit flies like bananas Input Output Fruit flies like bananas NP/NP NP ( S \ NP ) /NP NP > > NP S \ NP < S like bananas Fruit flies ( S \ NP ) /NP NP NP/NP NP > > NP S \ NP Fruit like flies bananas NP/NP NP ( S \ NP ) /NP NP ∅ 9
Parsing with Hypergraphs Fruit flies like bananas Input Output Fruit flies like bananas Fruit flies like bananas NP NP \ NP ( S \ NP ) /NP NP NP/NP NP ( S \ NP ) /NP NP > > < > NP S \ NP NP S \ NP < < S S like bananas Fruit flies like bananas Fruit flies ( S \ NP ) /NP NP NP NP \ NP ( S \ NP ) /NP NP NP/NP NP > < > > NP S \ NP NP S \ NP Fruit flies like bananas Fruit like flies bananas NP NP \ NP ( S \ NP ) /NP NP NP/NP NP ( S \ NP ) /NP NP ∅ ∅ 10
Parsing with Hypergraphs Fruit flies like bananas Input Output Fruit flies like bananas Fruit flies like bananas NP NP \ NP ( S \ NP ) /NP NP NP/NP NP ( S \ NP ) /NP NP > > < > NP S \ NP NP S \ NP < < S S like bananas Fruit flies like bananas Each hyperedge is Fruit flies e ( S \ NP ) /NP NP NP NP \ NP ( S \ NP ) /NP NP NP/NP NP > g ( e ) < weighted with a score > > NP S \ NP NP S \ NP Fruit flies like bananas Fruit like flies bananas NP NP \ NP ( S \ NP ) /NP NP NP/NP NP ( S \ NP ) /NP NP ∅ ∅ 11
Parsing with Hypergraphs Fruit flies like bananas Input Output Fruit flies like bananas Fruit flies like bananas NP NP \ NP ( S \ NP ) /NP NP NP/NP NP ( S \ NP ) /NP NP > > < > NP S \ NP NP S \ NP < < S S Score of parse derivation: X like bananas Fruit flies g ( y ) = g ( e ) like bananas Fruit flies ( S \ NP ) /NP NP NP NP \ NP ( S \ NP ) /NP NP NP/NP NP > < > > NP S \ NP NP S \ NP e ∈ y Fruit flies like bananas Fruit like flies bananas NP NP \ NP ( S \ NP ) /NP NP NP/NP NP ( S \ NP ) /NP NP ∅ ∅ 12
Parsing with Hypergraphs Fruit flies like bananas Fruit flies like bananas NP NP \ NP ( S \ NP ) /NP NP NP/NP NP ( S \ NP ) /NP NP < > > > NP S \ NP NP S \ NP < < S S Fruit flies Fruit flies like bananas ( S \ NP ) /NP NP NP NP \ NP NP/NP NP > < > NP NP S \ NP flies Fruit like flies bananas NP \ NP NP/NP NP ( S \ NP ) /NP NP ∅ Fruit flies like NP S \ NP ( S \ S ) /NP like bananas Fruit flies NP S \ NP ( S \ S ) /NP NP Fruit flies like bananas < > S S \ S NP S \ NP ( S \ S ) /NP NP < > S S \ S < S 13
Parsing with Hypergraphs Fruit flies like bananas Fruit flies like bananas NP/NP NP ( S \ NP ) /NP NP NP NP \ NP ( S \ NP ) /NP NP > > < > NP S \ NP NP S \ NP < < S S Fruit flies like bananas Fruit flies like bananas NP NP \ NP ( S \ NP ) /NP NP NP/NP NP ( S \ NP ) /NP NP < > > > NP S \ NP NP S \ NP < < S S Fruit flies Fruit flies like bananas ( S \ NP ) /NP NP NP NP \ NP NP/NP NP > < > NP NP S \ NP flies Fruit like flies bananas NP \ NP NP/NP NP ( S \ NP ) /NP NP ∅ Fruit flies like NP S \ NP ( S \ S ) /NP like bananas Fruit flies NP S \ NP ( S \ S ) /NP NP Fruit flies like bananas < > S S \ S NP S \ NP ( S \ S ) /NP NP < > S S \ S < S 14
Parsing with Hypergraphs Fruit flies like bananas Fruit flies like bananas NP NP \ NP ( S \ NP ) /NP NP NP/NP NP ( S \ NP ) /NP NP < > > > NP S \ NP NP S \ NP < < S S y ∗ = argmax ❖ Predicted parse: Fruit flies Fruit flies like bananas g ( y ) ( S \ NP ) /NP NP NP NP \ NP NP/NP NP > < > NP NP S \ NP y ∈ Y ❖ Exponential number of nodes flies Fruit like flies bananas NP \ NP NP/NP NP ( S \ NP ) /NP NP ∅ Fruit flies like Intractable inference NP S \ NP ( S \ S ) /NP like bananas Fruit flies NP S \ NP ( S \ S ) /NP NP Fruit flies like bananas < > S S \ S NP S \ NP ( S \ S ) /NP NP < > S S \ S < S 15
Managing Intractable Search Spaces Approximate inference with Fruit flies like bananas Fruit flies like bananas NP NP \ NP ( S \ NP ) /NP NP NP/NP NP ( S \ NP ) /NP NP global expressivity , e.g. < > > > NP S \ NP NP S \ NP < < S S ❖ Greedy / beam search: Fruit flies Fruit flies like bananas ( S \ NP ) /NP NP NP NP \ NP NP/NP NP ❖ Nivre, 2008 > < > NP NP S \ NP ❖ Chen and Manning, 2014 flies Fruit like flies bananas ❖ Andor et al., 2016 NP \ NP NP/NP NP ( S \ NP ) /NP NP ∅ ❖ Reranking: Fruit flies like NP S \ NP ( S \ S ) /NP ❖ Charniak and Johnson, 2005 like bananas Fruit flies NP S \ NP ( S \ S ) /NP NP Fruit flies like bananas ❖ Huang, 2008 < > S S \ S NP S \ NP ( S \ S ) /NP NP < > S S \ S < ❖ Socher et al., 2013 S 16
Locally Factored Parsing Scores condition on local structures ❖ Make locality assumptions: Fruit flies like bananas ? S ❖ e.g. features are local to CFG like bananas Fruit flies ? ? NP S \ NP productions flies Fruit flies like bananas NP \ NP NP/NP NP ( S \ NP ) /NP NP ❖ Polynomial number of nodes ∅ ❖ Dynamic programs enable flies like Fruit NP S \ NP ( S \ S ) /NP tractable inference like bananas Fruit flies ? ? S S \ S 17
Locally Factored Parsing Scores condition on local structures Dynamic programs with Fruit flies like bananas ? S locally factored models , e.g. like bananas Fruit flies ? ? NP S \ NP ❖ CKY: ❖ Collins, 1997 flies Fruit flies like bananas ❖ Durrett and Klein, 2015 NP \ NP NP/NP NP ( S \ NP ) /NP NP ∅ ❖ Minimum spanning tree: flies like Fruit NP S \ NP ( S \ S ) /NP ❖ McDonald et al., 2005 ❖ Kiperwasser and Goldberg, 2016 like bananas Fruit flies ? ? S S \ S 18
Locally Factored Parsing Scores condition on local structures Dynamic programs with Fruit flies like bananas ? S locally factored models , e.g. like bananas Fruit flies ? ? NP S \ NP ❖ CKY: Recursive neural networks ❖ Collins, 1997 flies Fruit flies like bananas ❖ Durrett and Klein, 2015 NP \ NP NP/NP NP ( S \ NP ) /NP NP break dynamic programs! ∅ ❖ Minimum spanning tree: flies like Fruit NP S \ NP ( S \ S ) /NP ❖ McDonald et al., 2005 ❖ Kiperwasser and Goldberg, 2016 like bananas Fruit flies ? ? S S \ S 19
Recommend
More recommend
