Lecture 16: The CKY parsing algorithm Kai-Wei Chang CS @ - PowerPoint PPT Presentation

Lecture 16: The CKY parsing algorithm Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse webpage: http://kwchang.net/teaching/NLP16 CS6501: NLP 1

How to represent the structure CS6501: NLP 2

Phrase structure (constituency) trees v Can be modeled by Context-free grammars v We will see how constituent parse and dependency parse are related CS6501: NLP 3

Parse tree defined by CFG 1 2 3 4 5 6 7 CS6501: NLP 4

Naïve top-down parsing v # possible trees is exponential v Many sub-trees are the same CS6501: NLP 5

Two key issues v Computational complexity v Can we reuse the computations? Dynamic programming v Ambiguity v (Lexicalized) PCFG CS6501: NLP 6

Chomsky Normal Form v Chomsky Normal Form allows only two kinds of right-hand sides: v Two non-terminals: (e.g., VP → ADV VP) v One terminal: VP → eat v Any CFG can be rewritten into an equivalent Chomsky Normal Form CS6501: NLP 7

Chomsky Normal Form v Chomsky Normal Form allows only two kinds of right-hand sides: v Two non-terminals: (e.g., VP → ADV VP) v One terminal: VP → eat v Any CFG can be rewritten into an equivalent Chomsky Normal Form v Try this: VP → VBD NP PP PP CS6501: NLP 8

More about the conversion v Eliminate rules with non-solitary terminals 𝐵 → 𝑌 $ … 𝑏 … 𝑌 ( add Y * → 𝑏 and replace the rule with 𝐵 → 𝑌 $ … 𝑍 * … 𝑌 ( , Y * → 𝑏 v Eliminate right-hand sides with > 2 nonterminals 𝐵 → 𝑌 $ … 𝑌 . … 𝑌 ( replace the rule by 𝐵 → 𝑌 $ 𝐵 $ ,𝐵 $ → 𝑌 0 𝐵 0 ,… , 𝐵 (10 → 𝑌 (1$ 𝑌 ( v Remove unit rules 𝐵 → 𝐶 replace each rule 𝐶 → 𝑌 $ 𝑌 0 with A → 𝑌 $ 𝑌 0 CS6501: NLP 9

One more example v Example from https://en.wikipedia.org/wiki/Chomsky_normal_form <S> → Expr Expr → Term | Expr AddOp Term | AddOp Term Term → Factor | Term MulOp Factor Factor → Primary | Factor ^ Primary Primar → number | variable | ( Expr ) y AddOp → + | − MulOp → * | / CS6501: NLP 10

Let’s try to parse! “Papa ate the caviar with a spoon” 0 1 2 3 4 5 6 7 NP à Papa N à caviar S à NP VP N à spoon NP à Det N V à spoon NP à NP PP V à ate VP à V NP P à with VP à VP PP Det à the PP à P NP Det à a The following slides are modified from Jason Eisner’s NLP course CS6501: NLP 11

§ initialize the list with parts-of-speech (non-terminal) “Papa ate the caviar with a spoon” NP V Det N P Det N § for each constituent on the LIST § for each adjacent constituent on the list § for each rule to combine them § add the result to the LIST if it’s not already there § if the above loop added anything, do it again! CS6501: NLP 12

“Papa ate the caviar with a spoon” 0 1 2 3 4 5 6 7 Initialize 1st pass 2nd pass 3rd pass § NP 0 1 § NP 2 4 § VP 1 4 § … § V 1 2 § NP 5 7 § NP 2 4 § Det 2 3 § PP 4 7 § N 3 4 § NP 5 7 § P 4 5 § Det 5 6 § N 6 7 § V 6 7 CS6501: NLP 13

This is correct, but we repeatedly check same pairs! CS6501: NLP 14

What are the problems? We kept checking the same pairs that we’d checked before (both bad and good pairs) Can’t we manage the process in a way that avoids duplicate work? And even finding new pairs was expensive because we had to scan the whole list Can’t we have some kind of index that will help us find adjacent pairs? CS6501: NLP 15

Indexing: Store S 0 4 in chart[0,4] S à NP VP NP à Det N NP à NP PP VP à V NP VP à VP PP PP à P NP Every cell stores constituents found between i and j CS6501: NLP 16

Avoid duplicate work: S à NP VP NP à Det N NP à NP PP VP à V NP VP à VP PP PP à P NP CS6501: NLP 17

S à NP VP How to build a width-6 phrase NP à Det N NP à NP PP VP à V NP VP à VP PP PP à P NP ? 1 7 = 1 2 + 2 7 J 1 3 + 3 7 1 4 + 4 7 J 1 5 + 5 7 1 6 + 6 7 CS6501: NLP 18

CKY algorithm § for J := 1 to n § Add to [J-1,J] all categories for the J th word § for width := 2 to n § for start := 0 to n-width // this is I § Define end := start + width // this is J § for mid := start+1 to end-1 // find all I-to-J phrases § for every nonterminal Y in [start,mid] § for every nonterminal Z in [mid,end] § for all nonterminals X § if X à Y Z is in the grammar § then add X to [start,end] CS6501: NLP 19

CKY algorithm § for J := 1 to n § Add to [J-1,J] all categories for the J th word § for width := 2 to n § for start := 0 to n-width // this is I § Define end := start + width // this is J § for mid := start+1 to end-1 // find all I-to-J phrases § for every rule X à Y Z in the grammar if Y in [start,mid] and Z in [mid,end] then add X to [start,end] CS6501: NLP 20

CKY parsing algorithm v Dynamic programming: v Save the results in a table and reuse the computations v Complexity: 𝑃 𝑜 6 𝐻 𝑜: length of the sentence, |𝐻| : size of grammar v Assume CFG is in Chomsky Normal Form CS6501: NLP 21

Your turn! 𝑇 → 𝑂𝑄 𝑊𝑄 𝑊𝑄 → 𝑊 𝑂𝑄 𝑊𝑄 → 𝑊𝑄 𝑄𝑄 𝑄𝑄 → 𝑄 𝑂𝑄 𝑂𝑄 → 𝑂𝑄 𝑄𝑄 NP V NP P NP CS6501: NLP 22

𝑇 → 𝑂𝑄 𝑊𝑄 𝑊𝑄 → 𝑊 𝑂𝑄 𝑊𝑄 → 𝑊𝑄 𝑄𝑄 𝑄𝑄 → 𝑄 𝑂𝑄 𝑂𝑄 → 𝑂𝑄 𝑄𝑄 𝑇 CS6501: NLP 23

Probability Context-Free Grammars CS6501: NLP 24

Likelihood of a parse tree CS6501: NLP 25

Probabilistic CKY CS6501: NLP 26

time 1 flies 2 like 3 an 4 arrow 5 NP 3 NOTE that the in the following animation Vst 3 we consider “minimizing the loss”. The algorithm that “maximizes the probability” will be similar. 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 1 VP → V NP 1 VP 4 2 VP → VP PP 1 NP → Det N P 2 2 2 NP → NP PP V 5 3 NP → NP NP 3 Det 1 4 0 PP → P NP N 8 The following slides are modified from Jason Eisner’s NLP course CS6501: NLP 27

time 1 flies 2 like 3 an 4 arrow 5 NP 3 Vst 3 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 1 VP → V NP 1 VP 4 2 VP → VP PP 1 NP → Det N P 2 2 2 NP → NP PP V 5 3 NP → NP NP 3 Det 1 4 0 PP → P NP N 8 CS6501: NLP 28

time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 1 VP → V NP 1 VP 4 2 VP → VP PP 1 NP → Det N P 2 2 2 NP → NP PP V 5 3 NP → NP NP 3 Det 1 4 0 PP → P NP N 8

time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 1 VP → V NP 1 VP 4 2 VP → VP PP 1 NP → Det N P 2 2 2 NP → NP PP V 5 3 NP → NP NP 3 Det 1 4 0 PP → P NP N 8

time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S 13 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 1 VP → V NP 1 VP 4 2 VP → VP PP 1 NP → Det N P 2 2 2 NP → NP PP V 5 3 NP → NP NP 3 Det 1 4 0 PP → P NP N 8

time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S 13 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 1 VP → V NP 1 VP 4 2 VP → VP PP 1 NP → Det N P 2 2 2 NP → NP PP V 5 3 NP → NP NP 3 Det 1 4 0 PP → P NP N 8

time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S 13 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 1 VP → V NP 1 VP 4 2 VP → VP PP 1 NP → Det N P 2 2 2 NP → NP PP V 5 3 NP → NP NP 3 Det 1 NP 10 4 0 PP → P NP N 8

time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S 13 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 1 VP → V NP 1 VP 4 2 VP → VP PP 1 NP → Det N P 2 2 2 NP → NP PP V 5 3 NP → NP NP 3 Det 1 NP 10 4 0 PP → P NP N 8

time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S 13 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 1 VP → V NP 1 VP 4 2 VP → VP PP 1 NP → Det N P 2 PP 12 2 2 NP → NP PP V 5 3 NP → NP NP 3 Det 1 NP 10 4 0 PP → P NP N 8

time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S 13 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 1 VP → V NP 1 VP 4 2 VP → VP PP 1 NP → Det N P 2 PP 12 2 2 NP → NP PP V 5 VP 16 3 NP → NP NP 3 Det 1 NP 10 4 0 PP → P NP N 8

time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S 13 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 1 VP → V NP 1 VP 4 2 VP → VP PP 1 NP → Det N P 2 PP 12 2 2 NP → NP PP V 5 VP 16 3 NP → NP NP 3 Det 1 NP 10 4 0 PP → P NP N 8

time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S 13 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 NP 18 1 VP → V NP 1 VP 4 2 VP → VP PP 1 NP → Det N P 2 PP 12 2 2 NP → NP PP V 5 VP 16 3 NP → NP NP 3 Det 1 NP 10 4 0 PP → P NP N 8

time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S 13 0 1 S → NP VP 6 S → Vst NP 2 S → S PP NP 4 NP 18 1 VP → V NP 1 VP 4 S 21 2 VP → VP PP 1 NP → Det N P 2 PP 12 2 2 NP → NP PP V 5 VP 16 3 NP → NP NP 3 Det 1 NP 10 4 0 PP → P NP N 8