Sharing Across Expressions N N N N -ity -ity Adj -ness -ity Adj Adj Adj -able -able -able V V V -able V agree agree agree count Thursday, February 2, 2012
Remarks on Inference Tradeoff • Nothing fancy here. • The two simplicity biases are just Bayesian prior and likelihood applied to computation and storage problem. • Lexicon code length and data code length given lexicon in (two part) MDL . • Can be connected with many other frameworks.
Inference as Conditioning • Inference Process: Probabilistic Conditioning. • Define joint model. P(Data, Fragments) = P(Data | Fragments) * P(Fragments) 60
Inference as Conditioning • Inference Process: Probabilistic Conditioning. Likelihood (derivation probabilities) • Define joint model. P(Data, Fragments) = P(Data | Fragments) * P(Fragments) 61
Inference as Conditioning Prior • Inference Process: Probabilistic Conditioning. (lexicon probabilities) • Define joint model. P(Data, Fragments) = P(Data | Fragments) * P(Fragments) 62
Inference as Conditioning • Inference Process: Probabilistic Conditioning. • Condition on particular dataset. P(Fragments | Data) ∝ P(Data | Fragments) * P(Fragments) 63
Probabilistic Conditioning • Intuition: two-step algorithm. 1. Throw away lexicons not consistent with the data. 2. Renormalize remaining lexicons so that they sum to one. • Maximally conservative: Relative beliefs are always conserved. 64
The Mathematical Model: Fragment Grammars • Generalization of Adaptor Grammars (Johnson et al., 2007). • Allows storing of partial trees. • Framework first proposed in MDL setting by De Marcken, 1996. • Related to work on probabilistic tree- substitution grammars (e.g., Bod, 2003; Cohn, 2010; Goodman, 2003; Zuidema, 2007; Post, 2013).
Talk Outline 1. Introduction to productivity and reuse with Fragment Grammars (with Noah Goodman). 2. Case Studies on Productivity and Competition.
Case Studies • Other approaches to productivity and reuse. 1. What distributions signal productivity? 2. How is competition resolved? 3. Multi-way competition.
Case Studies • Other approaches to productivity and reuse. 1. What distributions signal productivity? 2. How is competition resolved? 3. Multi-way competition.
Four Strategies for Productivity and Reuse Thursday, February 2, 2012
Four Strategies for Productivity and Reuse • 5 Formal Models Thursday, February 2, 2012
Four Strategies for Productivity and Reuse • 5 Formal Models • Capture historical proposals from the literature. Thursday, February 2, 2012
Four Strategies for Productivity and Reuse • 5 Formal Models • Capture historical proposals from the literature. • Minimally different. Thursday, February 2, 2012
Four Strategies for Productivity and Reuse • 5 Formal Models • Capture historical proposals from the literature. • Minimally different. • Same inputs, same underlying space of representations. Thursday, February 2, 2012
Four Strategies for Productivity and Reuse • 5 Formal Models • Capture historical proposals from the literature. • Minimally different. • Same inputs, same underlying space of representations. • State-of-the-art probabilistic models. Thursday, February 2, 2012
Full-Parsing Full-Parsing (FP) (MAP Multinomial-Dirichlet Context- Free Grammars ) - All generalizations are productive. - Minimal abstract units. - Johnson, et al. 2007a - Estimated on token frequency. N N N N -ity -ity Adj -ness -ity Adj Adj Adj -able -able -able V V V -able V agree agree agree count
Full-Listing N N N N Full-Parsing -ity -ity Adj Adj -ness Adj -ity Adj (FP) -able -able V V -able V -able V (MAP All-Adapted Adaptor agree agree agree count Grammars ) Full-Listing - Store whole form after first use (FL) (recursively). - Maximally specific units. - Johnson, et al. 2007 - Base system estimated on type frequencies. - Formalization of classical lexical redundancy rules. N N N N -ity Adj -ity -ness -ity Adj Adj Adj -able V -able -able V V -able V agree agree agree count
N N N N Exemplar-Based Full-Parsing -ity -ity Adj Adj -ness Adj -ity Adj (FP) -able -able V V -able V -able V agree agree agree count ( Data-Oriented Parsing ) N N N N -ity Adj Full-Listing -ity -ness -ity Adj Adj Adj - (FL) Store all generalizations -able V -able V -able V -able V agree consistent with input. agree agree count - Two Formalization: Data-Oriented Exemplar-Based Parsing 1 (DOP1; Bod, 1998) , Data- (EB) Oriented Parsing: Equal-Node Estimator (ENDOP; Goodman, 2003) . - Argued to be exemplar model of syntax. N N N N -ity -ity Adj -ness -ity Adj Adj Adj -able -able -able V V V -able V agree agree agree count
N N N N Inference-Based Full-Parsing -ity -ity Adj Adj -ness Adj -ity Adj (FP) -able -able V V -able V -able V agree agree agree count ( Fragment Grammars ) N N N N -ity Adj Full-Listing -ity -ness -ity Adj Adj Adj - (FL) Store set of subcomputations -able V -able V -able V -able V which best explains the data. agree agree agree count - N N N N Formalization: Fragment Exemplar-Based -ity -ity Adj Adj -ness Adj -ity Adj Grammars (O’Donnell, et al. 2009) (EB) -able -able V V -able V -able V - agree agree agree Inference depends on count distribution of tokens over types. - Inference-Based Only model which infers (IB) variables. N N N N -ity -ity Adj -ness -ity Adj Adj Adj -able -able -able V V V -able V agree agree agree count
Empirical Domains Past Tense Derivational (Inflectional) Morphology Productive +ed (walked) +ness (goodness) I → æ (sang) Context-Dependent +ity (ability) suppletion +th (width) Unproductive (go/went)
Case Studies • Other approaches to productivity and reuse. 1. What distributions signal productivity? 2. How is competition resolved? 3. Multi-way competition.
Empirical Evaluations Derivational Past Tense Morphology Productive +ed (walked) +ness (goodness) I → æ (sang) Context-Dependent +ity (ability) suppletion +th (width) Unproductive (go/went)
What (Distributional) Cues Signal Productivity? • Many proposals in the literature: • Type frequency. • Token frequency (combined with something else, e.g., entropy). • Heterogeneity of context (generalized type frequency).
Top 5 Most Productive Suffixes Full-Parsing (MDPCFG) Full-Listing (MAG) Inference-Based (FG) Exemplar (DOP1) Exemplar (ENDOP)
Top 5 Most Productive Suffixes Full-Parsing (MDPCFG) Full-Listing (MAG) Inference-Based (FG) Exemplar (DOP1) Exemplar (ENDOP)
Top 5 Most Productive Suffixes Full-Parsing (MDPCFG) Full-Listing (MAG) Inference-Based (FG) Exemplar (DOP1) Exemplar (GDMN)
What Evidences Productivity? • Crucial evidence of productivity: Use of a lexical item (morpheme, rule, etc.) to generate new forms. • Distributional consequence: Large proportion of low frequency forms.
What Predicts Productivity?
Top 5 Most Productive Suffixes Full-Parsing (MDPCFG) Full-Listing (MAG) High Proportion of Low Frequency Types Inference-Based (FG) Exemplar (DOP1) Exemplar (GDMN)
Top 5 Most Productive Suffixes High Type Frequency Full-Parsing (MDPCFG) Full-Listing (MAG) High Token Frequency Inference-Based (FG) High Token Frequency High Token Frequency Exemplar (DOP1) Exemplar (GDMN)
Baayen’s Hapax -Based Measures • Baayen’s / (e.g., Baayen, 1992) P P ∗ • Estimators of productivity based on the proportion of frequency- 1 words in an input corpus. • Various derivations. • Rate of vocabulary change in urn model. • Good-Turing estimation. • Fundamentally, a rule-of-thumb. • Only defined for single affix estimation.
Productivity Correlations ( values from Hay & Baayen, 2002) P / P ∗ MDPCFG (Exemplar-based) ENDOP DOP1 FG MAG (Exemplar-based) (Inference) (Full-parsing) (Full-listing)
Fragment Grammars and Hapaxes • For the case of single affixes, Fragments Grammars behave approximately as if they were using hapaxes. • Not an explicit assumption of the model • Model is about how words are built. Given the fact that some new words are built, behavior arises automatically. • Generalizes to multi-way competition.
Case Studies • Other approaches to productivity and reuse. 1. What distributions signal productivity? 2. How is competition resolved? 3. Multi-way competition.
Empirical Domains Derivational Past Tense Morphology Productive +ed (walked) +ness (goodness) I → æ (sang) Context-Dependent +ity (ability) suppletion +th (width) Unproductive (go/went)
Crucial Facts • Defaultness : Regular rule applies when all else fails. • Blocking : Existence of irregular blocks regular rule. • In this domain preferences are sharp.
How can Correct Inflection be Represented? Irregulars Regulars Thursday, February 2, 2012
How can Correct Inflection be Represented? Irregulars Regulars Thursday, February 2, 2012
Correct Inflection Irregular Regular Unattested 8 ( ( ( Log Odds Correct 6 Full-Parsing FP (Multinomial-Dirichlet CFG) 4 Full-Listing FL (Adaptor Grammars) Exemplar 2 E1 (Data-Oriented Parsing 1) Exemplar E2 0 (DOP: ENDOP) FPFL E1 E2 IB FP FL E1 E2 IB FP FL E1 E2 IB Inference-Based IB (Fragment Grammars) − 2 − 4 98
Correct Inflection Irregular Regular Unattested Preference 8 ( ( ( for Correct Log Odds Correct 6 Full-Parsing Past Form FP (Multinomial-Dirichlet CFG) 4 Full-Listing FL (Adaptor Grammars) Exemplar 2 E1 (Data-Oriented Parsing 1) Exemplar E2 0 (DOP: ENDOP) FPFL E1 E2 IB FP FL E1 E2 IB FP FL E1 E2 IB Inference-Based IB (Fragment Grammars) − 2 − 4 99
Correct Inflection Irregular Regular Unattested 8 ( ( ( Log Odds Correct 6 Full-Parsing FP (Multinomial-Dirichlet CFG) 4 Full-Listing FL (Adaptor Grammars) Exemplar 2 E1 (Data-Oriented Parsing 1) Exemplar E2 0 (DOP: ENDOP) FPFL E1 E2 IB FP FL E1 E2 IB FP FL E1 E2 IB Inference-Based IB (Fragment Grammars) − 2 − 4 Preference for Incorrect Past Form 100
Recommend
More recommend
