Transformational Priors The Big Concept Over Grammars Want to - PDF document

Transformational Priors The Big Concept Over Grammars � Want to parse (or build a syntactic language model). � Must estimate rule probabilities. � Problem: Too many possible rules! � Especially with lexicalization and flattening (which help). Jason Eisner Jason Eisner � So it’s hard to estimate probabilities. Johns Hopkins University July 6, 2002 — EMNLP The Big Concept Problem: Too Many Rules NP → DT fund 26 ... NN → fund � Problem: Too many rules! 24 NP → DT NN fund 8 � Especially with lexicalization and flattening (which help). S NNP → 7 fund S → 5 TO fund NP � So it’s hard to estimate probabilities. NP → NNP fund 2 NP → DT NPR NN fund 2 S → NP fund � Solution: Related rules tend to have related probs 2 TO fund NP PP TO TO NP → 1 DT JJ NN fund NP → 1 DT NPR JJ fund to � POSSIBLE relationships are given a priori NP → 1 DT ADJP NNP fund projects SBAR SBAR NP → 1 DT JJ JJ NN fund NP → 1 DT NN fund SBAR NPR → � LEARN which relationships are strong in this language 1 fund that ... NP-PRD → 1 DT NN fund VP NP → 1 DT NN fund PP (just like feature selection) NP → 1 DT ADJP NN fund ADJP NP → 1 DT ADJP fund PP NP → 1 DT JJ fund PP-TMP NP-PRD → 1 DT ADJP NN fund VP � Method has connections to: NP → 1 NNP fund , VP , NP → 1 PRP$ fund S-ADV → 1 DT JJ fund � Parameterized finite-state machines (Monday’s talk) NP → 1 DT NNP NNP fund SBAR → 1 NP MD fund NP PP NP → � Bayesian networks (inference, abduction, explaining away) 1 DT JJ JJ fund SBAR NP → 1 DT JJ NN fund SBAR NP → 1 DT NNP fund � Linguistic theory (transformations, metarules, etc.) NP → 1 NP$ JJ NN fund NP → 1 DT JJ fund Too Many Rules … But Luckily … [Want To Multiply Rule Probabilities] NP → DT fund NP → DT fund 26 26 ... ... NN → fund NN → fund 24 24 NP → DT NN fund NP → DT NN fund 8 8 S NNP → S NNP → 7 fund 7 fund S → S → 5 TO fund NP 5 TO fund NP NP → NNP fund NP → NNP fund 2 2 NP → DT NPR NN fund NP → DT NPR NN fund 2 2 S → S → NP NP fund fund 2 TO fund NP PP 2 TO fund NP PP TO TO NP TO TO NP → NP → 1 DT JJ NN fund 1 DT JJ NN fund NP → NP → 1 DT NPR JJ fund 1 DT NPR JJ fund to to NP → NP → 1 DT ADJP NNP fund 1 DT ADJP NNP fund projects SBAR SBAR NP → projects SBAR SBAR NP → 1 DT JJ JJ NN fund 1 DT JJ JJ NN fund NP → NP → 1 DT NN fund SBAR 1 DT NN fund SBAR NPR → NPR → 1 fund 1 fund NP-PRD → NP-PRD → that ... that ... 1 DT NN fund VP 1 DT NN fund VP NP → NP → 1 DT NN fund PP 1 DT NN fund PP NP → NP → 1 DT ADJP NN fund ADJP 1 DT ADJP NN fund ADJP NP → NP → 1 DT ADJP fund PP 1 DT ADJP fund PP NP → NP → 1 DT JJ fund PP-TMP 1 DT JJ fund PP-TMP NP-PRD → NP-PRD → 1 DT ADJP NN fund VP 1 DT ADJP NN fund VP All these rules for fund – NP → NP → p(tree) = ... p( | S) × p( | TO) × p( | NP) × p( | SBAR) × ... 1 NNP fund , VP , 1 NNP fund , VP , NP → NP → 1 PRP$ fund 1 PRP$ fund S-ADV → S-ADV → 1 DT JJ fund & other, still unobserved rules – 1 DT JJ fund NP → NP → 1 DT NNP NNP fund 1 DT NNP NNP fund SBAR → SBAR → (oversimplified) 1 NP MD fund NP PP 1 NP MD fund NP PP NP → are connected by the deep NP → 1 DT JJ JJ fund SBAR 1 DT JJ JJ fund SBAR NP → NP → 1 DT JJ NN fund SBAR 1 DT JJ NN fund SBAR NP → NP → 1 DT NNP fund 1 DT NNP fund NP → structure of English. NP → 1 NP$ JJ NN fund 1 NP$ JJ NN fund NP → NP → 1 DT JJ fund 1 DT JJ fund 1

Rules Are Related Rules Are Related NP → DT fund NP → DT fund 26 26 NN → fund NN → fund 24 24 � fund behaves like a � fund behaves like a NP → DT NN fund NP → DT NN fund 8 8 typical singular noun … NNP → typical singular noun … NNP → 7 fund 7 fund S → S → 5 TO fund NP 5 TO fund NP � … or transitive verb … NP → NNP fund NP → NNP fund one fact! 2 2 NP → DT NPR NN fund NP → DT NPR NN fund 2 2 though PCFG represents it as many apparently unrelated rules. S → S → one more fact! 2 TO fund NP PP 2 TO fund NP PP NP → NP → 1 DT JJ NN fund 1 DT JJ NN fund even if several more rules. NP → NP → 1 DT NPR JJ fund 1 DT NPR JJ fund NP → NP → 1 DT ADJP NNP fund 1 DT ADJP NNP fund NP → NP → 1 DT JJ JJ NN fund 1 DT JJ JJ NN fund Verb rules are RELATED . NP → NP → 1 DT NN fund SBAR 1 DT NN fund SBAR NPR → NPR → 1 fund ... 1 fund NP-PRD → NP-PRD → 1 DT NN fund VP Should be able to PREDI CT the ones we haven’t seen. 1 DT NN fund VP NP → NP → 1 DT NN fund PP 1 DT NN fund PP NP → NP → 1 DT ADJP NN fund ADJP 1 DT ADJP NN fund ADJP NP → NP → 1 DT ADJP fund PP 1 DT ADJP fund PP S NP → NP → 1 DT JJ fund PP-TMP 1 DT JJ fund PP-TMP NP-PRD → NP-PRD → 1 DT ADJP NN fund VP 1 DT ADJP NN fund VP NP → NP → 1 NNP fund , VP , 1 NNP fund , VP , NP → NP → 1 PRP$ fund 1 PRP$ fund S-ADV → S-ADV → 1 DT JJ fund 1 DT JJ fund NP → NP → 1 DT NNP NNP fund 1 DT NNP NNP fund NP SBAR → SBAR → fund 1 NP MD fund NP PP TO TO 1 NP MD fund NP PP NP → 1 DT JJ JJ fund SBAR NP → NP → 1 DT JJ JJ fund SBAR 1 DT JJ NN fund SBAR NP → NP → to 1 DT JJ NN fund SBAR 1 DT NNP fund NP → NP → projects SBAR SBAR 1 DT NNP fund 1 NP$ JJ NN fund NP → NP → 1 NP$ JJ NN fund 1 DT JJ fund NP → 1 DT JJ fund that ... Rules Are Related Rules Are Related NP → DT fund NP → DT fund 26 26 NN → fund NN → fund � fund behaves like a 24 � fund behaves like a 24 NP → DT NN fund NP → DT NN fund 8 8 typical singular noun … NNP → typical singular noun … NNP → 7 fund 7 fund S → S → 5 TO fund NP 5 TO fund NP � … or transitive verb … � … or transitive verb … NP → NNP fund NP → NNP fund 2 2 NP → DT NPR NN fund NP → DT NPR NN fund 2 2 S → S → � … but as noun, has an � … but as noun, has an 2 TO fund NP PP 2 TO fund NP PP NP → NP → 1 DT JJ NN fund 1 DT JJ NN fund idiosyncratic fondness NP → idiosyncratic fondness NP → 1 DT NPR JJ fund 1 DT NPR JJ fund NSF issued the grant NP → NP → 1 DT ADJP NNP fund 1 DT ADJP NNP fund NP → NP → 1 DT JJ JJ NN fund 1 DT JJ JJ NN fund NP → NP → for purpose clauses … 1 DT NN fund SBAR for purpose clauses … 1 DT NN fund SBAR The grant issued today NPR → NPR → 1 fund 1 fund NP-PRD → DT NN fund VP 1 NP-PRD → 1 DT NN fund VP NP → 1 DT NN fund PP NP → 1 DT NN fund PP NP → 1 DT ADJP NN fund ADJP ??? NP → 1 DT ADJP NN fund ADJP � … and maybe other NP → 1 DT ADJP fund PP NP → 1 DT ADJP fund PP NP → 1 DT JJ fund PP-TMP NP → 1 DT JJ fund PP-TMP NP-PRD → NP-PRD → DT ADJP NN fund VP 1 DT ADJP NN fund VP 1 idiosyncrasies to be NP → 1 NNP fund , VP , NP → NSF funded the grant NP → the ACL fund to put proceedings online 1 NNP fund , VP , 1 PRP$ fund S-ADV → 1 DT JJ fund NP → discovered, like NP → 1 PRP$ fund 1 DT NNP NNP fund The grant funded today S-ADV → SBAR → the old ACL fund for students to attend ACL 1 DT JJ fund 1 NP MD fund NP PP NP → NP → 1 DT NNP NNP fund 1 DT JJ JJ fund SBAR SBAR → unaccusativity … NP → 1 NP MD fund NP PP 1 DT JJ NN fund SBAR NP → one more fact! NP → 1 DT JJ JJ fund SBAR unlikely sentence, but if we do see it, 1 DT NNP fund NP → NP → 1 NP$ JJ NN fund 1 DT JJ NN fund SBAR NP → 1 DT JJ fund is unaccusativity plausible ? (vs. other parse) predicts dozens of unseen rules NP → 1 DT NNP fund NP → 1 NP$ JJ NN fund All This Is Quantitative! Format of the Rules NP → DT fund 26 S S NN → fund � fund behaves like a 24 NP → DT NN fund 8 typical singular noun … NNP → 7 fund NP VP S → 5 TO fund NP NP put NP PP � … or transitive verb … NP → NNP fund 2 Jim NP → DT NPR NN fund 2 VP PP S → Jim pizza in the oven � … but as noun, has an 2 TO fund NP PP NP → 1 DT JJ NN fund how often? idiosyncratic fondness NP → in the oven 1 DT NPR JJ fund NP → 1 DT ADJP NNP fund V NP NP → 1 DT JJ JJ NN fund NP → for purpose clauses … 1 DT NN fund SBAR NPR → 1 fund NP-PRD → 1 DT NN fund VP and how NP → put pizza 1 DT NN fund PP NP → 1 DT ADJP NN fund ADJP � … and maybe other NP → 1 DT ADJP fund PP NP → 1 DT JJ fund PP-TMP S → NP put NP PP → NP VP NP-PRD → does that 1 DT ADJP NN fund VP (put) S idiosyncrasies to be NP → 1 NNP fund , VP , NP → 1 PRP$ fund VP → VP PP S-ADV → (put) 1 DT JJ fund discovered, like NP → tell us 1 DT NNP NNP fund VP → V NP SBAR → 1 NP MD fund NP PP (put) NP → 1 DT JJ JJ fund SBAR unaccusativity … NP → 1 DT JJ NN fund SBAR → put NP → 1 DT NNP fund V (put) p(rule)? NP → 1 NP$ JJ NN fund NP → 1 DT JJ fund 2