A comparison of selectional preference models for automatic verb - PowerPoint PPT Presentation

Introduction Models Results References A comparison of selectional preference models for automatic verb classification Will Roberts and Markus Egg Institut für Anglistik und Amerikanistik Humboldt Universität zu Berlin Sunday, 26 October, 2014 Will Roberts Selprefs for verb classification 1 / 20

Introduction Models Results References Outline Introduction 1 Models 2 Results 3 Will Roberts Selprefs for verb classification 2 / 20

Introduction Models Results References Selectional preferences Predicates can select for their arguments: ? My aunt is a bachelor. (McCawley, 1968) We model verbs empirically: I eat meat bread fruit . . . newspaper Evaluate on an automatic verb classification task Baseline model clusters verbs based on subcategorisation Will Roberts Selprefs for verb classification 3 / 20

Introduction Models Results References Selectional preferences Example Wir benutzen Ihre Umfragedaten nicht für eigene Zwecke. We use your survey data not for own purposes. We will not use your survey responses for private purposes. We will want to record that this instance of use has: Subject wir, we (pronoun, ignored) Direct object Umfragedatum, survey datum PP (für, for ) Zweck, purpose We also include indirect objects (datives) A selectional preference model will map noun forms onto concept labels Will Roberts Selprefs for verb classification 4 / 20

Introduction Models Results References Hypothesis effective SP model verb clustering score ineffective SP model only subcat: optimal lexical one concept concept preferences: containing granularity one concept all nouns per noun Will Roberts Selprefs for verb classification 5 / 20

Introduction Models Results References Subcategorisation Example Wir Ihre Umfragedaten nicht für eigene Zwecke. benutzen We use your survey data not for own purposes. We will not use your survey responses for private purposes. The combination of syntactic argument types is assigned a subcategorisation frame (SCF) code: benutzen ⇒ nap:für.Acc A verb’s distribution over SCF codes is its subcategorisation preference Will Roberts Selprefs for verb classification 6 / 20

Introduction Models Results References Pipeline test set hierarchical SdeWaC mate-tools SCF verb gold corpus dependency tagger clustering clusters standard parser (Ward’s) selectional preferences model Test set has 3 million verb instances Gold standard: 168 verbs in 43 classes Will Roberts Selprefs for verb classification 7 / 20

Introduction Models Results References Verb clustering scf 671 scf 672 scf 673 scf 1 scf 2 scf 3 scf 4 . . . verb: 6 12 3 7 2 12 11 corpus counts p = 1 discrete probability distribution p = 0 = subcat prefs Verb dissimilarity is computed with the Jensen-Shannon divergence Will Roberts Selprefs for verb classification 8 / 20

Introduction Models Results References Lexical preferences (LP) Example Wir benutzen Ihre Umfragedaten nicht für eigene Zwecke. We use your survey data not for own purposes. We will not use your survey responses for private purposes. benutzen ⇒ nap:für.Acc*dobj-Umfragedatum*prep-Zweck To control data sparsity, we employ a parameter N : number of nouns included in the lexical preferences model Nouns with rank > N are ignored (as if unseen) Will Roberts Selprefs for verb classification 9 / 20

Introduction Models Results References Sun/Korhonen � verb N , dative � � verb N , subj � � verb 2 , prep � � verb 1 , subj � � verb 2 , subj � � verb N , obj � � verb 1 , obj � . . . noun: 3 8 4 10 4 7 11 corpus counts discrete p = 1 probability distribution p = 0 Partition N nouns into M classes (equivalence relation) Will Roberts Selprefs for verb classification 10 / 20

Introduction Models Results References Word space model (WSM) Built on lemmatised SdeWaC Features are the 50,000 most common words (minus stop words) Sentences as windows Feature weighting: t-test scheme Context selection zeroes out infrequent features in the model Use cosine similarity and spectral clustering to partition N nouns into M classes Will Roberts Selprefs for verb classification 11 / 20

Introduction Models Results References GermaNet target set, depth ≤ 1 GNROOT_n_1 Granularity is controlled Stelle_n_1 Menge_n_2 Entitaet_n_2 using depth , d 0.5 0.375 0.125 Nouns can belong to more Jahr_n_1 kognitives_Objekt_n_1 than one concept: soft clustering Zeitabschnitt_n_1 zyklische_Zeiteinheit_n_1 Jahr_n_2 Will Roberts Selprefs for verb classification 12 / 20

Introduction Models Results References Latent Dirichlet Allocation (LDA) α Built with the same data used by the Φ Sun/Korhonen model Each � verb , grammatical relation � pair β z has a distribution Φ over concepts Each concept z has a distribution Θ over the N nouns Θ W Number of concepts M is 50 or 100 M n G Will Roberts Selprefs for verb classification 13 / 20

Introduction Models Results References Results SP model Parameters Granularity F -score SUN 10K nouns 1,000 noun classes 39.76 LDA (hard) 10K nouns 50 topics 39.09 LP 5K nouns 38.02 WSM 10K nouns 500 noun classes 36.92 LDA (soft) 10K nouns 50 topics 35.91 GermaNet depth = 5 8,196 synsets 34.41 Baseline 33.47 Will Roberts Selprefs for verb classification 14 / 20

Introduction Models Results References Sparsity effects in LP 38 0.6 37 0.5 36 0.4 Coverage 35 PairF 0.3 34 0.2 33 0.1 32 31 0.0 10 0 10 1 10 2 10 3 10 4 10 5 N Will Roberts Selprefs for verb classification 15 / 20

Introduction Models Results References Qualitative differences in noun partitions SUN WSM F -score 39.76 F -score 36.92 syntagmatic information paradigmatic information synonym/co-hyponym structure thematic structure class size variance 37 class size variance 2800 semantically consistent large classes inconsistent Will Roberts Selprefs for verb classification 16 / 20

Introduction Models Results References Test set size 45 40 35 PairF 30 25 Baseline sun lda -hard lp 20 wsm 15 10 5 10 6 10 7 10 8 Number of verb instances Will Roberts Selprefs for verb classification 17 / 20

Introduction Models Results References Conclusions 1 Selectional preferences help automatic verb classification 2 Optimal concept granularity is relatively fine Lexical preferences works very well if it is properly tuned Classification of proper names is useful: given names, corporations, medications, etc. 3 Syntagmatic information works better than paradigmatic Will Roberts Selprefs for verb classification 18 / 20

Introduction Models Results References Summary Selectional preference models have been compared before Almost always under a plausibility or pseudoword paradigm! We are interested in semantic verb clustering We evaluate several selectional preference models, comparing them using a manually constructed semantic verb classification We show that modelling selectional preferences is beneficial for verb clustering, no matter which selectional preference model we choose Other findings: Capturing syntagmatic relations seems to work better than paradigmatic A simple lexical preferences model performs very well; data sparsity does not seem to be more of a problem for this model than for others Will Roberts Selprefs for verb classification 19 / 20

Introduction Models Results References References James D. McCawley. The role of semantics in a grammar. In Emmon Bach and Robert Harms, editors, Universals in Linguistic Theory , pages 124–169. Holt, Rinehart and Winston, 1968. Will Roberts Selprefs for verb classification 20 / 20