Dynamic Programming for Linear Time Incremental Parsing Liang Huang Information Sciences Institute University of Southern California Kenji Sagae Institute for Creative Technologies University of Southern California ACL 2010, Uppsala, Sweden, July 2010 (slightly expanded)
Ambiguities in Parsing I feed cats nearby in the garden ... • let’s focus on dependency structures for simplicity • ambiguous attachments of nearby and in • ambiguity explodes exponentially with sentence length • must design efficient (polynomial) search algorithm • typically using dynamic programming (DP); e.g. CKY DP for Incremental Parsing (Huang and Sagae) 2
Ambiguities in Parsing I feed cats nearby in the garden ... • let’s focus on dependency structures for simplicity • ambiguous attachments of nearby and in • ambiguity explodes exponentially with sentence length • must design efficient (polynomial) search algorithm • typically using dynamic programming (DP); e.g. CKY DP for Incremental Parsing (Huang and Sagae) 2
Ambiguities in Parsing I feed cats nearby in the garden ... • let’s focus on dependency structures for simplicity • ambiguous attachments of nearby and in • ambiguity explodes exponentially with sentence length • must design efficient (polynomial) search algorithm • typically using dynamic programming (DP); e.g. CKY DP for Incremental Parsing (Huang and Sagae) 2
Ambiguities in Parsing I feed cats nearby in the garden ... • let’s focus on dependency structures for simplicity • ambiguous attachments of nearby and in • ambiguity explodes exponentially with sentence length • must design efficient (polynomial) search algorithm • typically using dynamic programming (DP); e.g. CKY DP for Incremental Parsing (Huang and Sagae) 2
But full DP is too slow... I feed cats nearby in the garden ... • full DP (like CKY) is too slow (cubic-time) • while human parsing is fast & incremental (linear-time) DP for Incremental Parsing (Huang and Sagae) 3
But full DP is too slow... I feed cats nearby in the garden ... • full DP (like CKY) is too slow (cubic-time) • while human parsing is fast & incremental (linear-time) • how about incremental parsing then? • yes, but only with greedy search (accuracy suffers) • explores tiny fraction of trees (even w/ beam search) DP for Incremental Parsing (Huang and Sagae) 3
But full DP is too slow... I feed cats nearby in the garden ... • full DP (like CKY) is too slow (cubic-time) • while human parsing is fast & incremental (linear-time) • how about incremental parsing then? • yes, but only with greedy search (accuracy suffers) • explores tiny fraction of trees (even w/ beam search) • can we combine the merits of both approaches? • a fast, incremental parser with dynamic programming? • explores exponentially many trees in linear-time? DP for Incremental Parsing (Huang and Sagae) 3
Linear-Time Incremental DP incremental parsing greedy (e.g. shift-reduce) search ☹ (Nivre 04; Collins/Roark 04; ...) this work: full DP principled fast shift-reduce parsing (e.g. CKY) search with dynamic programming ☺ (Eisner 96; Collins 99; ...) ☹ ☺ fast slow (linear-time) (cubic-time) DP for Incremental Parsing (Huang and Sagae) 4
Preview of the Results • very fast linear-time dynamic programming parser • best reported dependency accuracy on PTB/CTB • explores exponentially many trees (and outputs forest) 1.4 parsing time (secs) 1.2 1 0.8 0.6 0.4 0.2 0 0 10 20 30 40 50 60 70 sentence length DP for Incremental Parsing (Huang and Sagae) 5
Preview of the Results • very fast linear-time dynamic programming parser • best reported dependency accuracy on PTB/CTB • explores exponentially many trees (and outputs forest) 1.4 y k e parsing time (secs) a l i e 1.2 n k r r a 1 e h B C MST 0.8 0.6 0.4 this work 0.2 0 0 10 20 30 40 50 60 70 sentence length DP for Incremental Parsing (Huang and Sagae) 5
Preview of the Results • very fast linear-time dynamic programming parser • best reported dependency accuracy on PTB/CTB • explores exponentially many trees (and outputs forest) 1.4 y 10 10 DP: exponential k e parsing time (secs) a l i e 1.2 n k 10 8 r r a 1 e h B 10 6 C MST 0.8 0.6 10 4 0.4 10 2 this work 0.2 non-DP beam search 10 0 0 0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70 sentence length sentence length DP for Incremental Parsing (Huang and Sagae) 5
Outline • Motivation • Incremental (Shift-Reduce) Parsing • Dynamic Programming for Incremental Parsing • Experiments DP for Incremental Parsing (Huang and Sagae) 6
Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - feed cats nearby ... cats nearby in ... cats nearby in ... nearby in the ... nearby in the ... in the garden ... DP for Incremental Parsing (Huang and Sagae) 7
Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift cats nearby in ... cats nearby in ... nearby in the ... nearby in the ... in the garden ... DP for Incremental Parsing (Huang and Sagae) 8
Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... cats nearby in ... nearby in the ... nearby in the ... in the garden ... DP for Incremental Parsing (Huang and Sagae) 9
Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I nearby in the ... nearby in the ... in the garden ... DP for Incremental Parsing (Huang and Sagae) 10
Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I 4 shift feed cats nearby in the ... I nearby in the ... in the garden ... DP for Incremental Parsing (Huang and Sagae) 11
Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I 4 shift feed cats nearby in the ... I 5a r-reduce feed nearby in the ... cats I in the garden ... DP for Incremental Parsing (Huang and Sagae) 12
Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I 4 shift feed cats nearby in the ... I 5a r-reduce feed nearby in the ... cats I 5b shift in the garden ... feed cats nearby I DP for Incremental Parsing (Huang and Sagae) 13
Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I 4 shift feed cats nearby in the ... shift-reduce I 5a r-reduce feed conflict nearby in the ... cats I 5b shift in the garden ... feed cats nearby I DP for Incremental Parsing (Huang and Sagae) 14
Choosing Parser Actions ← stack queue → ← stack queue → in the garden ... ... feed cats ... s 2 s 1 s 0 q 0 q 1 ... nearby I features: (s 0 . w , s 0 . rc , q 0 , ...) = (cats, nearby, in, ...) • score each action using features f and weights w • features are drawn from a local window • abstraction (or signature) of a state -- this inspires DP! • weights trained by structured perceptron (Collins 02) DP for Incremental Parsing (Huang and Sagae) 15
Greedy Search • each state => three new states (shift, l-reduce, r-reduce) • search space should be exponential • greedy search: always pick the best next state DP for Incremental Parsing (Huang and Sagae) 16
Greedy Search • each state => three new states (shift, l-reduce, r-reduce) • search space should be exponential • greedy search: always pick the best next state DP for Incremental Parsing (Huang and Sagae) 17
Beam Search • each state => three new states (shift, l-reduce, r-reduce) • search space should be exponential • beam search: always keep top- b states DP for Incremental Parsing (Huang and Sagae) 18
Dynamic Programming • each state => three new states (shift, l-reduce, r-reduce) • key idea of DP: share common subproblems • merge equivalent states => polynomial space DP for Incremental Parsing (Huang and Sagae) 19
Dynamic Programming • each state => three new states (shift, l-reduce, r-reduce) • key idea of DP: share common subproblems • merge equivalent states => polynomial space “graph-structured stack” (Tomita, 1988) DP for Incremental Parsing (Huang and Sagae) 20
Dynamic Programming • each state => three new states (shift, l-reduce, r-reduce) • key idea of DP: share common subproblems • merge equivalent states => polynomial space “graph-structured stack” (Tomita, 1988) DP for Incremental Parsing (Huang and Sagae) 21
Recommend
More recommend
