learning word representations by jointly modeling
play

Learning Word Representations by Jointly Modeling Syntagmatic and - PowerPoint PPT Presentation

Feb 08, 2023 •589 likes •1.02k views

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun , Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng ofey.sunfei@gmail.com CAS Key Lab of Network Data Science and Technology Institute of Computing

  1. Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun , Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng ofey.sunfei@gmail.com CAS Key Lab of Network Data Science and Technology Institute of Computing Technology, Chinese Academy of Sciences October 22, 2015 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 1 / 27

  2. Word Representations Sentiment Machine Analysis Translation [Kalchbrenner and [Maas et al., Blunsom, 2013] 2011] language POS Word Taging modeling Representations [Collobert et al., [Bengio et al., 2011] 2003] Word-Sense Parsing Disambiguation [Socher et al., [Collobert et al., 2011] 2011] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 2 / 27

  3. Word Representations Models GloVe [Pennington et al., 2014] Word2Vec [Mikolov et al., 2013] LBL [Mnih and Hinton, 2007] NPLMs [Bengio et al., 2003] LDA [Blei et al., 2003] HAL [Deerwester et al., 1990] LSI BOW [Lund et al., 1995] [Harris, 1954] 1940 1960 1980 2000 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 3 / 27

  4. Word Representations Models GloVe [Pennington et al., 2014] Word2Vec [Mikolov et al., 2013] LBL [Mnih and Hinton, 2007] NPLMs [Bengio et al., 2003] Relations? LDA [Blei et al., 2003] HAL [Deerwester et al., 1990] LSI BOW [Lund et al., 1995] [Harris, 1954] 1940 1960 1980 2000 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 3 / 27

  5. Relations One Hypothesis Two Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 4 / 27

  6. The Distributional Hypothesis [Harris, 1954, Firth, 1957] “ You shall know a word by the company it keeps .” —J.R. Firth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 5 / 27

  7. Syntagmatic and Paradigmatic Relations [Sahlgren, 2008] syntagmatic was a physicist. Albert Einstein paradigmatic Feynman was a physicist. Richard syntagmatic • Syntagmatic: words co-occur in the same text region • Paradigmatic: words occur in the same context, may not at the same time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 6 / 27

  8. Syntagmatic syntagmatic was a physicist. Albert Einstein Feynman was a physicist. Richard syntagmatic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 7 / 27

  9. Syntagmatic syntagmatic was a physicist. Albert Einstein Feynman was a physicist. Richard syntagmatic d 1 d 2 Einstein 1 0 Feynman 0 1 physicist 1 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 7 / 27

  10. Syntagmatic syntagmatic was a physicist. Albert Einstein Feynman was a physicist. Richard syntagmatic Feynman physicist 1 d 1 d 2 Einstein 1 0 Feynman 0 1 physicist 1 1 Einstein 0 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 7 / 27

  11. Syntagmatic syntagmatic was a physicist. Albert Einstein Feynman was a physicist. Richard syntagmatic Feynman physicist 1 d 1 d 2 Einstein 1 0 Feynman 0 1 physicist 1 1 Einstein 0 1 LSI, LDA, PV-DBOW · · · . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 7 / 27

  12. Paradigmatic was a physicist. Albert Einstein paradigmatic Feynman was a physicist. Richard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 8 / 27

  13. Paradigmatic was a physicist. Albert Einstein paradigmatic Feynman was a physicist. Richard Einstein Feynman physicist Einstein 0 0 1 Feynman 0 0 1 physicist 1 1 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 8 / 27

  14. Paradigmatic was a physicist. Albert Einstein paradigmatic Feynman was a physicist. Richard Einstein Feynman physicist Einstein 0 0 1 Einstein Feynman Feynman 0 0 1 physicist physicist 1 1 0 0 0 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 8 / 27

  15. Paradigmatic was a physicist. Albert Einstein paradigmatic Feynman was a physicist. Richard Einstein Feynman physicist Einstein 0 0 1 Einstein Feynman Feynman 0 0 1 physicist physicist 1 1 0 0 0 0 NLMs, Word2Vec, GloVe · · · . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 8 / 27

  16. Motivation syntagmatic was a physicist. Albert Einstein paradigmatic Feynman was a physicist. Richard syntagmatic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 9 / 27

  17. Motivation syntagmatic was a physicist. Albert Einstein paradigmatic Word2Vec Feynman was a physicist. Richard syntagmatic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 9 / 27

  18. Motivation syntagmatic was a physicist. Albert Einstein paradigmatic Word2Vec PV-DBOW Feynman was a physicist. Richard syntagmatic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 9 / 27

  19. Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 10 / 27

  20. nw nw Parallel Document Context Model (PDC) . . N . ∑ ∑ log p ( w n i | h n ℓ = i ) the w n n =1 i ∈ d n Paradigmatic cat on the sat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 11 / 27

  21. nw nw Parallel Document Context Model (PDC) . . N . ( ) ∑ ∑ log p ( w n i | h n i )+ log p ( w n ℓ = i | d n ) the w n n =1 i ∈ d n Paradigmatic cat on the sat . . . . . . the cat sat Syntagmatic on the . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 11 / 27

  22. Parallel Document Context Model (PDC) . . N . ( ) ∑ ∑ log p ( w n i | h n i )+ log p ( w n ℓ = i | d n ) the w n n =1 i ∈ d n Paradigmatic Negative Sampling cat N ( ∑ ∑ log σ ( ⃗ i · ⃗ i )+ log σ ( ⃗ i · ⃗ w n h n w n ℓ = d n ) w n n =1 i ∈ d n on w ′ · ⃗ + k · E w ′ ∼ P nw log σ ( − ⃗ h n i ) ) w ′ · ⃗ the + k · E w ′ ∼ P nw log σ ( − ⃗ d n ) sat 1 . σ ( x ) = . . 1 + exp ( − x ) . . . the cat sat Syntagmatic on the . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 11 / 27

  23. Parallel Document Context Model (PDC) . . N . ( ) ∑ ∑ log p ( w n i | h n i )+ log p ( w n ℓ = i | d n ) the w n n =1 i ∈ d n Paradigmatic Negative Sampling cat N ( ∑ ∑ log σ ( ⃗ i · ⃗ i )+ log σ ( ⃗ i · ⃗ w n h n w n ℓ = d n ) w n n =1 i ∈ d n on w ′ · ⃗ + k · E w ′ ∼ P nw log σ ( − ⃗ h n i ) ) w ′ · ⃗ the + k · E w ′ ∼ P nw log σ ( − ⃗ d n ) sat 1 . σ ( x ) = . . 1 + exp ( − x ) PDC PV-DM . . . the cat MF for W-D + W-C sat not clear Syntagmatic on the . . . [Levy and Goldberg, 2014] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 11 / 27

  24. nw nc Hierarchical Document Context Model (HDC) N ∑ ∑ log p ( w n ℓ = i | d n ) w n n =1 i ∈ d n sat Syntagmatic . . . the cat sat on the . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION October 22, 2015 12 / 27

Recommend

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun ,

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun ,

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun , Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng CAS Key Lab of Network Data Science and Technology Institute of Computing Technology, Chinese

524 views • 40 slides
Jointly Learning Word Representations and Composition Functions Using Predicate-Argument

Jointly Learning Word Representations and Composition Functions Using Predicate-Argument

Jointly Learning Word Representations and Composition Functions Using Predicate-Argument Structures Kazuma Hashimoto (UT) Pontus Stenetorp (UT) Makoto Miwa (TTI) Yoshimasa Tsuruoka (UT) U niversity of T okyo ( UT ) T oyota T echnological I

1.39k views • 119 slides
Inside Out: Two Jointly Predictive Models for Word Representations and Phrase Representations Fei

Inside Out: Two Jointly Predictive Models for Word Representations and Phrase Representations Fei

Inside Out: Two Jointly Predictive Models for Word Representations and Phrase Representations Fei Sun , Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng ofey.sunfei@gmail.com CAS Key Lab of Network Data Science and Technology Institute of

701 views • 37 slides
Jointly Learning Word and Phrase Embeddings Using Neural Networks and Implicit Tensor

Jointly Learning Word and Phrase Embeddings Using Neural Networks and Implicit Tensor

Jointly Learning Word and Phrase Embeddings Using Neural Networks and Implicit Tensor Factorization Kazuma Hashimoto Tsuruoka Laboratory, University of Tokyo 19/06/2015 Talk@UCL Machine Reading Lab. Self Introduction Name Kazuma

726 views • 39 slides
NLU lecture 5: Word representations and morphology Adam Lopez alopez@inf.ed.ac.uk Essential

NLU lecture 5: Word representations and morphology Adam Lopez alopez@inf.ed.ac.uk Essential

NLU lecture 5: Word representations and morphology Adam Lopez alopez@inf.ed.ac.uk Essential epistemology Word representations and word2vec Word representations and compositional morphology Reading: Mikolov et al. 2013, Luong et al.

746 views • 42 slides
Incorporating Relational Knowledge into Word Representations using Subspace Regularization Jun

Incorporating Relational Knowledge into Word Representations using Subspace Regularization Jun

Incorporating Relational Knowledge into Word Representations using Subspace Regularization Jun Araki (Carnegie Mellon University) joint work with Abhishek Kumar (IBM Research) ACL 2016 Distributed word representations Low-dimensional dense

378 views • 18 slides
Contextual Word Representations with BERT and Other Pre-trained Language Models Jacob Devlin

Contextual Word Representations with BERT and Other Pre-trained Language Models Jacob Devlin

Contextual Word Representations with BERT and Other Pre-trained Language Models Jacob Devlin Google AI Language History and Background Pre-training in NLP Word embeddings are the basis of deep learning for NLP king queen [-0.5, -0.9,

852 views • 43 slides
Batch IS NOT Heavy: Learning Word Representations From All Samples 1 1 1 Xin Xin, Fajie Yuan,

Batch IS NOT Heavy: Learning Word Representations From All Samples 1 1 1 Xin Xin, Fajie Yuan,

Batch IS NOT Heavy: Learning Word Representations From All Samples 1 1 1 Xin Xin, Fajie Yuan, Xiangnan He, Joemon Jose 2 1 School of Computing Science, University of Glasgow 2 School of Computing, National University of Singapore Presented by

603 views • 28 slides
Word representations Benoit Favre < benoit.favre@univ-mrs.fr > Aix-Marseille Universit,

Word representations Benoit Favre < benoit.favre@univ-mrs.fr > Aix-Marseille Universit,

Deep learning for natural language processing Word representations Benoit Favre < benoit.favre@univ-mrs.fr > Aix-Marseille Universit, LIF/CNRS 21 Feb 2017 Benoit Favre (AMU) DL4NLP: word embeddings 21 Feb 2017 1 / 26 Deep learning

432 views • 26 slides
Word Meaning: Distributional Representations & Word Sense Disambiguation CMSC 723 / LING 723

Word Meaning: Distributional Representations & Word Sense Disambiguation CMSC 723 / LING 723

Word Meaning: Distributional Representations & Word Sense Disambiguation CMSC 723 / LING 723 / INST 725 Marine Carpuat Slides credit: Dan Jurafsky Reminders Read the syllabus Make sure you have access to piazza Get started on

691 views • 54 slides
Learning Representations for Multi-Dimensional Sentiment Andrew Maas, Peter Pham, Ying Wang Word

Learning Representations for Multi-Dimensional Sentiment Andrew Maas, Peter Pham, Ying Wang Word

Learning Representations for Multi-Dimensional Sentiment Andrew Maas, Peter Pham, Ying Wang Word Vectors Maas et al (ACL 2011) The Experience Project this is sad, i just roasted marshmallow. by myself. on the stove top. lol. at least they

72 views • 6 slides
Sentence and Contextualised Word Representations Graham Neubig Site

Sentence and Contextualised Word Representations Graham Neubig Site

CS11-747 Neural Networks for NLP Sentence and Contextualised Word Representations Graham Neubig Site https://phontron.com/class/nn4nlp2019/ (w/ slides by Antonis Anastasopoulos) Sentence Representations We can create a vector or sequence

619 views • 43 slides
Lesson 10 Deep learning for NLP: Mul6lingual Word Sequence Modeling December 15, 2016 EPFL

Lesson 10 Deep learning for NLP: Mul6lingual Word Sequence Modeling December 15, 2016 EPFL

Human Language Technology: Applica6on to Informa6on Access Lesson 10 Deep learning for NLP: Mul6lingual Word Sequence Modeling December 15, 2016 EPFL Doctoral Course EE-724 Nikolaos Pappas Idiap Research Ins6tute, Mar6gny Outline of the

1.6k views • 86 slides
Using Machine Learning to Study the Neural Representations of Language Meanings Tom M. Mitchell

Using Machine Learning to Study the Neural Representations of Language Meanings Tom M. Mitchell

Using Machine Learning to Study the Neural Representations of Language Meanings Tom M. Mitchell Carnegie Mellon University June 2017 How does neural activity encode word meanings? How does neural activity encode word meanings? How does

673 views • 64 slides
Representations of Power Series over Word Algebras Ulrich Faigle, Alexander Schnhuth

Representations of Power Series over Word Algebras Ulrich Faigle, Alexander Schnhuth

Guideline Introduction Representations Solutions Conclusion Representations of Power Series over Word Algebras Ulrich Faigle, Alexander Schnhuth Mathematical Institute University Cologne Centrum Wiskunde & Informatica Amsterdam CTW

534 views • 39 slides
Word representations and modelling ambiguity: A case study of metaphor Ekaterina Shutova ILLC

Word representations and modelling ambiguity: A case study of metaphor Ekaterina Shutova ILLC

Word representations and modelling ambiguity: A case study of metaphor Ekaterina Shutova ILLC University of Amsterdam 2 May 2018 Word representations and modelling ambiguity: A case study of metaphor 1 / 29 Polysemy and word senses The

790 views • 29 slides
Deep Learning for Natural Language Processing More training methods for word embeddings Richard

Deep Learning for Natural Language Processing More training methods for word embeddings Richard

Deep Learning for Natural Language Processing More training methods for word embeddings Richard Johansson richard.johansson@gu.se overview research on vector-based word representations goes back to the 1990s but took of in 2013 with the

404 views • 14 slides
Word, Sense and Contextualized Embeddings: Vector Representations of Meaning in NLP Jose

Word, Sense and Contextualized Embeddings: Vector Representations of Meaning in NLP Jose

Word, Sense and Contextualized Embeddings: Vector Representations of Meaning in NLP Jose Camacho-Collados Cardiff University, 18 March 2019 1 Outline Background Vector Space Models (word embeddings) Lexical resources Sense

1.33k views • 98 slides
Restrictive Learning with Distributions over Underlying Representations Karen Jesney, Joe Pater

Restrictive Learning with Distributions over Underlying Representations Karen Jesney, Joe Pater

Restrictive Learning with Distributions over Underlying Representations Karen Jesney, Joe Pater & Robert Staubs University of Massachusetts Amherst {kjesney, pater, rstaubs} @ linguist.umass.edu Workshop on Computational Modeling of Sound

603 views • 47 slides
Lexical representations in theories of word reading Stphan Tulkens Overview Color legend

Lexical representations in theories of word reading Stphan Tulkens Overview Color legend

Lexical representations in theories of word reading Stphan Tulkens Overview Color legend 1. Word reading a. Introduction ENGLISH b. Common sense definition DUTCH 2. The dictionary metaphor FRENCH a. The mental

530 views • 32 slides
Modeling Uncertainty using Accept & Reject Statements Erik Quaeghebeur (much jointly with

Modeling Uncertainty using Accept & Reject Statements Erik Quaeghebeur (much jointly with

Modeling Uncertainty using Accept & Reject Statements Erik Quaeghebeur (much jointly with Gert de Cooman & Filip Hermans) Centrum Wiskunde & Informatica Amsterdam, the Netherlands The setup Experiment with outcomes in some

862 views • 41 slides
DeepWalk: Online Learning of Social Representations ACM SIG-KDD August 26, 2014 Bryan Perozzi ,

DeepWalk: Online Learning of Social Representations ACM SIG-KDD August 26, 2014 Bryan Perozzi ,

DeepWalk: Online Learning of Social Representations ACM SIG-KDD August 26, 2014 Bryan Perozzi , Rami Al-Rfou, Steven Skiena Stony Brook University Outline Introduction: Graphs as Features Language Modeling DeepWalk

528 views • 35 slides
A Probabilistic Model of Cross- situational Word Learning from Noisy and Ambiguous Data Afra

A Probabilistic Model of Cross- situational Word Learning from Noisy and Ambiguous Data Afra

A Probabilistic Model of Cross- situational Word Learning from Noisy and Ambiguous Data Afra Alishahi Joint work with Afsaneh Fazly and Suzanne Stevenson, University of Toronto 1 Word Learning Word learning: a mapping between a word and

885 views • 47 slides
Segmentation of Argumentative Texts with Contextualised Word Representations Georgios Petasis

Segmentation of Argumentative Texts with Contextualised Word Representations Georgios Petasis

Segmentation of Argumentative Texts with Contextualised Word Representations Georgios Petasis Software and Knowledge Engineering Laboratory, Institute of Informatics and Telecommunications, N.C.S.R. Demokritos , Athens, Greece

196 views • 17 slides

More recommend