PCFGs: Viterbi CKY CMSC 473/673 UMBC November 13 th , 2017
Recap from last time…
Probabilistic Context Free Grammar 1.0 S NP VP 1.0 PP P NP .4 NP Det Noun .34 AdjP Adj Noun .3 NP Noun .26 VP V NP .2 NP Det AdjP .0003 Noun Baltimore .1 NP NP PP … Set of weighted (probabilistic) rewrite Q: What are the distributions? rules, comprised of terminals and What must sum to 1? non-terminals Terminals: the words in the language (the lexicon), e.g., Baltimore A: P(X Y Z | X) Non-terminals: symbols that can trigger rewrite rules, e.g., S, NP , Noun (Sometimes) Pre-terminals: symbols that can only trigger lexical rewrites, e.g., Noun
Probabilistic Context Free Grammar S p( VP ) * NP S NP Noun p( ) * p( ) * p( )= NP VP Noun Baltimore VP NP Noun Verb Verb p( ) * p( ) * is Baltimore is a great city NP Verb product of probabilities of NP p( ) individual rules used in the derivation a great city
Probabilistic Context Free Grammar (PCFG) Tasks any Find the most likely parse (for an observed sequence) Calculate the (log) likelihood of an observed sequence w 1 , …, w N Learn the grammar parameters
CKY Precondition Grammar must be in Chomsky Normal Form (CNF) non-terminal non-terminal non-terminal X Y Z binary rules can only involve non-terminals non-terminal terminal X a unary rules can only involve terminals no ternary (+) rules
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” S NP VP NP Papa NP Det N N caviar Goal: NP NP PP N spoon VP V NP V spoon (S, 0, 7) VP VP PP V ate PP P NP P with Det the Entire grammar Assume uniform weights Det a Example from Jason Eisner
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” S NP VP NP Papa NP Det N N caviar NP NP PP N spoon First : Let’s find all NPs VP V NP V spoon VP VP PP V ate (NP, 0, 1): Papa (NP, 0, 1) (VP, 1, 7) (S, 0, 7) PP P NP P with (NP, 2, 4): the caviar (NP, 5, 7): a spoon Det the Entire grammar (NP, 2, 7): the caviar with a spoon Assume uniform Det a weights Second : Let’s find all VPs end 1 2 3 4 5 6 7 (VP, 1, 7): ate the caviar with a spoon 0 (VP, 1, 4): ate the caviar NP 1 VP Third : Let’s find all Ss 2 (S, 0, 7): Papa ate the caviar with a 3 start spoon 4 (S, 0, 4): Papa ate the caviar 5 6 Example from Jason Eisner
CKY Recognizer Input: * string of N words * grammar in CNF Output: True (with parse)/False Data structure: N*N table T Rows indicate span start (0 to N-1) Columns indicate span end (1 to N) T[i][j] lists constituents spanning i j
T = Cell[N][N+1] CKY Recognizer for(j = 1; j ≤ N; ++j) { T [j-1][j].add(X for non-terminal X in G if X word j ) } for(width = 2; width ≤ N; ++width ) { for(start = 0; start < N - width; ++start ) { end = start + width for(mid = start+1; mid < end; ++mid ) { for(non-terminal Y : T [start][mid ]) { Y Z for(non-terminal Z : T [mid][end ]) { T [start][end].add(X for rule X Y Z : G ) X } } Y Z } } }
T = Cell[N][N+1] CKY Recognizer for(j = 1; j ≤ N; ++j) { Q : What do we return? T [j-1][j].add(X for non-terminal X in G if X word j ) } for(width = 2; width ≤ N; ++width ) { A : S in T [0][N] for(start = 0; start < N - width; ++start ) { end = start + width for(mid = start+1; mid < end; ++mid ) { for(non-terminal Y : T [start][mid ]) { for(non-terminal Z : T [mid][end ]) { Q: How do we get the parse? T [start][end].add(X for rule X Y Z : G ) } } } A : Follow backpointers } }
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” S NP VP NP Papa NP Det N N caviar NP NP PP N spoon Work through VP V NP V spoon parse on board VP VP PP V ate PP P NP P with Det the Entire grammar Assume uniform weights Det a Example from Jason Eisner
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” 1 2 3 4 5 6 7 0 NP S S V VP VP 1 Det NP NP 2 N 3 P PP 4 Det NP 5 N 6 V Example from Jason Eisner
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” 1 2 3 4 5 6 7 0 NP S S V VP VP 1 Det NP NP 2 N 3 P PP 4 Det NP 5 N 6 V Example from Jason Eisner
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” 1 2 3 4 5 6 7 0 NP S S V VP VP 1 Det NP NP 2 N 3 P PP 4 Det NP 5 N 6 V Example from Jason Eisner
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” 1 2 3 4 5 6 7 0 NP S S V VP VP 1 Det NP NP 2 N 3 P PP 4 Det NP 5 N 6 V Example from Jason Eisner
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” 1 2 3 4 5 6 7 0 NP S S V VP VP 1 Det NP NP 2 N 3 P PP 4 Det NP 5 N 6 V Example from Jason Eisner
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” 1 2 3 4 5 6 7 0 NP S S V VP VP 1 Det NP NP 2 N 3 P PP 4 Det NP 5 N 6 V Example from Jason Eisner
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” 1 2 3 4 5 6 7 0 NP S S V VP VP 1 Det NP NP 2 N 3 P PP 4 Det NP 5 N 6 V Example from Jason Eisner
T = Cell[N][N+1] CKY Recognizer for(j = 1; j ≤ N; ++j) { T [j-1][j].add(X for non-terminal X in G if X word j ) } for(width = 2; width ≤ N; ++width ) { for(start = 0; start < N - width; ++start ) { end = start + width for(mid = start+1; mid < end; ++mid ) { for(non-terminal Y : T [start][mid ]) { for(non-terminal Z : T [mid][end ]) { T [start][end].add(X for rule X Y Z : G ) } } } } }
T = Cell[N][N+1] CKY Recognizer for(j = 1; j ≤ N; ++j) { T [j-1][j].add(X for non-terminal X in G if X word j ) } for(width = 2; width ≤ N; ++width ) { for(start = 0; start < N - width; ++start ) { end = start + width for(mid = start+1; mid < end; ++mid ) { for(rule X Y Z : G ) { T [start][end].add(X if Y in T [start][mid] & Z in T [mid][end]) } } } }
T = bool [K] [N][N+1] CKY Recognizer for(j = 1; j ≤ N; ++j) { for(non-terminal X in G if X word j ) { T[X] [j-1][j] = True } } for(width = 2; width ≤ N; ++width ) { for(start = 0; start < N - width; ++start ) { end = start + width for(mid = start+1; mid < end; ++mid ) { for(rule X Y Z : G ) { T[X] [start][end] = T[Y] [start][mid] & T[Z] [mid][end] } } } }
Probabilistic Context Free Grammar (PCFG) Tasks Find the most likely parse (for an observed sequence) Calculate the (log) likelihood of an observed sequence w 1 , …, w N Learn the grammar parameters
CKY Viterbi Parser Input: * string of N words * probabilistic grammar in CNF Output: Parse with probability (or None) Data structure: K*N*N table T K non-terminal symbols in the grammar Rows indicate span start (0 to N-1) Columns indicate span end (1 to N) T[X][i][j] lists most likely constituents beginning with rule X spanning i j
T = WeightedCell[K][N][N+1] CKY Viterbi for(j = 1; j ≤ N; ++j) { T [X][j-1][j] = argmax { p( X word j ) for non-terminal X in G if X word j } } for(width = 2; width ≤ N; ++width ) { for(start = 0; start < N - width; ++start ) { end = start + width for(mid = start+1; mid < end; ++mid ) { T [X][start][end] = argmax { p( X Y Z) * T [Y][start][mid] * T [Z][mid][end] for rule X Y Z : G } } } }
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” 1.0 S NP VP .1 NP Papa .6 NP Det N .6 N caviar .3 NP NP PP .4 N spoon .6 VP V NP .1 V spoon .4 VP VP PP .9 V ate 1.0 PP P NP 1.0 P with .5 Det the Entire grammar .5 Det a Example from Jason Eisner
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” 1 2 3 4 5 6 7 0 .1 NP S S .9 V VP VP 1 .5 Det NP NP 2 .6 N 3 .1 NP Papa 1.0 P PP 4 .6 N caviar 1.0 S NP VP .4 N spoon .6 NP Det N .5 Det NP 5 .1 V spoon .3 NP NP PP .9 V ate .6 VP V NP .4 N 1.0 P with 6 .4 VP VP PP .1 V .5 Det the 1.0 PP P NP .5 Det a Example from Jason Eisner
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” 1 2 3 4 5 6 7 0 .1 NP S S .6 NP Det N .9 V VP VP 1 .5 * .6 * .6 = .18 .5 Det .18 NP NP 2 .6 N 3 .1 NP Papa 1.0 P .12 PP 4 .6 N caviar 1.0 S NP VP .4 N spoon .6 NP Det N .5 Det .12 NP 5 .1 V spoon .3 NP NP PP .9 V ate .6 VP V NP .4 N 1.0 P with 6 .4 VP VP PP .1 V .5 Det the 1.0 PP P NP .5 Det a Example from Jason Eisner
0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” 1 2 3 4 5 6 7 0 .1 NP S S .0972 .9 V VP VP 1 .5 Det .18 NP NP 2 .0065 .6 N 3 .1 NP Papa 1.0 P .12 PP 4 .6 N caviar 1.0 S NP VP .4 N spoon .6 NP Det N .5 Det .12 NP 5 .1 V spoon .3 NP NP PP .9 V ate .6 VP V NP .4 N 1.0 P with 6 .4 VP VP PP .1 V .5 Det the 1.0 PP P NP .5 Det a Example from Jason Eisner
Recommend
More recommend
