jointly learning word and phrase embeddings using neural
play

Jointly Learning Word and Phrase Embeddings Using Neural Networks - PowerPoint PPT Presentation

Jan 15, 2023 •317 likes •726 views

Jointly Learning Word and Phrase Embeddings Using Neural Networks and Implicit Tensor Factorization Kazuma Hashimoto Tsuruoka Laboratory, University of Tokyo 19/06/2015 Talk@UCL Machine Reading Lab. Self Introduction Name Kazuma

  1. Jointly Learning Word and Phrase Embeddings Using Neural Networks and Implicit Tensor Factorization Kazuma Hashimoto Tsuruoka Laboratory, University of Tokyo 19/06/2015 Talk@UCL Machine Reading Lab.

  2. Self Introduction • Name – Kazuma Hashimoto ( 橋本 和真 in Japanese) – http://www.logos.t.u-tokyo.ac.jp/~hassy/ • Belong – Tsuruoka Laboratory, University of Tokyo • April 2015 – present Ph.D. student • April 2013 – March 2015 Master’s student – National Centre for Text Mining (NaCTeM) • Research Interest – Word/phrase/document embeddings and their applications 2 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  3. Today’s Agenda 1. Background – Word and Phrase Embeddings 2. Jointly Learning Word and Phrase Embeddings – General Idea 3. Our Methods Focusing on Transitive Verb Phrases – Word Prediction (EMNLP 2014) – Implicit Tensor Factorization (CVSC 2015) 4. Experiments and Results 5. Summary 3 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  4. Today’s Agenda 1. Background – Word and Phrase Embeddings 2. Jointly Learning Word and Phrase Embeddings – General Idea 3. Our Methods Focusing on Transitive Verb Phrases – Word Prediction (EMNLP 2014) – Implicit Tensor Factorization (CVSC 2015) 4. Experiments and Results 5. Summary 4 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  5. Assigning Vectors to Words • Word: String  Index  Vector • Why vectors? – Word similarities can be measured using distance metrics of the vectors (e.g., the cosine similarity) cause trigger cause disorder trigger disease disorder disease mouse animal rat animal mouse rat Embedding words in a vector space 5 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  6. Approaches to Word Representations • Two approaches using large corpora : (systematic comparison of them in Baroni+ (2014) ) – Count-based approach • e.g.) Reducing the dimension of word co- occurrence matrix using SVD – Prediction-based approach • e.g.) Predicting words from their contexts using neural networks • We focus on prediction-based approach – Why? 6 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  7. Learning Word Embeddings • Prediction-based approaches usually – parameterize the word embeddings – learn them based on co-occurrence statistics • Word embeddings appearing in similar contexts get close to each other ------ word prediction using the word embedding ------ text data … the prevalence of drunken driving and accidents caused by drinking … target SkipGram model (Mikolov+, 2013) in word2vec 7 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  8. Task-Oriented Word Embeddings • Learning word embeddings for relation classification – To appear at CoNLL 2015 (just advertising) 8 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  9. Beyond Word Embeddings • Treating phrases and sentences as well as words – gaining much attention recently! make pay payment money make payment pay money make payment pay money Embedding phrases in a vector space 9 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  10. Approaches to Phrase Embeddings • Element-wise addition/multiplication (Lapata+, 2010) – 𝑤 sentnce = 𝑗 𝑤 𝑥 𝑗 • Recursive autoencoders (Socher+, 2011; Hermann+, 2013) – Using parse trees – 𝑤 parent = 𝑔(𝑤 left child , 𝑤 right child ) • Tensor/matrix-based methods – 𝑤 adj noun = 𝑁 adj 𝑤(noun) (Baroni+, 2010) – 𝑁 verb = 𝑗,𝑘 𝑤 subj 𝑗 T 𝑤 obj 𝑘 (Grefenstette+, 2011) • 𝑁 subj, verb, obj = {𝑤 subj T 𝑤 obj } ∗ 𝑁(verb) • 𝑤 subj, verb, obj = 𝑁 verb 𝑤 obj ∗ 𝑤 subj (Kartsaklis+, 2012) 10 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  11. Which Word Embeddings are the Best? • Co-occurrence matrix + SVD • C&W (Collobert+, 2011) • RNNLM (Mikolov+, 2013) • SkipGram/CBOW (Mikolov+, 2013) • vLBL/ivLBL (Mnih+, 2013) • Dependency-based SkipGram (Levy+, 2014) • Glove (Pennington+, 2014) Which word embeddings should we use for which composition methods? Joint leaning 11 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  12. Today’s Agenda 1. Background – Word and Phrase Embeddings 2. Jointly Learning Word and Phrase Embeddings – General Idea 3. Our Methods Focusing on Transitive Verb Phrases – Word Prediction (EMNLP 2014) – Implicit Tensor Factorization (CVSC 2015) 4. Experiments and Results 5. Summary 12 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  13. Co-Occurrence Statistics of Phrases • Word co-occurrence statistics  word embeddings • How about phrase embeddings? – Phrase co-occurrence statistics! similar meanings? The businessman pays his monthly fee in yen similar contexts The importer made payment in his own domestic currency 13 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  14. How to Identify Phrase-Word Relations? • Using Predicate-Argument Structures (PAS) – Enju parer (Miyao+, 2008) • Analyzes relations between phrases and words NP arguments NP NP VP NP The importer made payment in his own domestic currency verb preposition predicates 14 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  15. Today’s Agenda 1. Background – Word and Phrase Embeddings 2. Jointly Learning Word and Phrase Embeddings – General Idea 3. Our Methods Focusing on Transitive Verb Phrases – Word Prediction (EMNLP 2014) – Implicit Tensor Factorization (CVSC 2015) 4. Experiments and Results 5. Summary 15 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  16. Why Transitive Verb Phrases? • Meanings of transitive verbs are affected by their arguments – e.g.) run, make, etc.  Good target to test composition models earn make money pay use make payment make use (of) make 16 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  17. Possible Application: Semantic Search • Embedding subject-verb-object tuples in a vector space – Semantic similarities between SVOs can be used! 17 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  18. Training Data from Large Corpora • Focusing on the role of prepositional adjuncts – Prepositional adjuncts complement meanings of verb phrases  should be useful parse ------ ------ simplification English Wikipedia, BNC, etc. How to model the relationships between predicates and arguments? 18 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  19. Today’s Agenda 1. Background – Word and Phrase Embeddings 2. Jointly Learning Word and Phrase Embeddings – General Idea 3. Our Methods Focusing on Transitive Verb Phrases – Word Prediction (EMNLP 2014) – Implicit Tensor Factorization (CVSC 2015) 4. Experiments and Results 5. Summary 19 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  20. Word Prediction Model (like word2vec) • Predicting words in predicate-argument tuples max(0, 1- s ( currency )+ s ( furniture ))  cost function currency furniture feature vector + for the word prediction prep ⊙ 𝐰 𝑏𝑠𝑕1 + 𝐪 = tanh(𝐢 𝑏𝑠𝑕1 prep prep arg1 pred 𝐢 𝑏𝑠𝑕1 𝐢 𝑞𝑠𝑓𝑒 prep ⊙ 𝐰 𝑞𝑠𝑓𝑒 ) 𝐢 𝑞𝑠𝑓𝑒 𝐰 𝑏𝑠𝑕1 𝐰 𝑞𝑠𝑓𝑒 PAS-CLBLM [importer make payment] in 20 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  21. How to Compute SVO Embeddings? • Two methods: – (a) assigning a vector to each SVO tuple – (b) composing SVO embeddings + - parameterized vectors - composed vectors subj verb obj [importer make payment] [importer make payment] (a) (b) 21 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  22. Today’s Agenda 1. Background – Word and Phrase Embeddings 2. Jointly Learning Word and Phrase Embeddings – General Idea 3. Our Methods Focusing on Transitive Verb Phrases – Word Prediction (EMNLP 2014) – Implicit Tensor Factorization (CVSC 2015) 4. Experiments and Results 5. Summary 22 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  23. Weakness of PAS-CLBLM • Only element-wise vector operations – Pros: Fast training – Cons: Poor interaction between predicates and arguments • Interactions between predicates and arguments are important for transitive verbs earn make money pay use make payment make use (of) make 23 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  24. Focusing on Tensor-Based Approaches • Tensor/matrix-based approaches (Noun: vector) – Adjective: matrix (Baroni+, 2010) – Transitive verb: matrix (Grefenstette+, 2011; Van de Cruys+, 2013) Given Given 𝑒 subject pre-trained 𝑒 ≅ subject verb 𝑒 verb Given 𝑄𝑁𝐽 (importer, make, payment) = 0.31 24 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  25. Implicit Tensor Factorization (1) • Parameterizing – Predicate matrices and – Argument embeddings Given Given argument 2 𝑒 𝑒 ≅ argument 2 predicate 𝑒 predicate Given 25 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  26. Implicit Tensor Factorization (2) • Calculating plausibility scores – Using predicate matrices & argument embeddings i k j 𝑈 ( i, j, k ) = Given Given argument 2 𝑒 𝑒 ≅ argument 2 predicate 𝑒 predicate Given 26 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

  27. Implicit Tensor Factorization (3) • Learning model parameters – Using plausibility judgment task • Observed tuple: ( i, j, k ) • Collapsed tuple: ( i ’ , j, k ), ( i, j’ , k ), ( i, j, k’ ) – Negative sampling (Mikolov+, 2013) Cost function 27 / 39 19/06/2015 Talk@UCL Machine Reading Lab.

Recommend

Word Embeddings Natural Language Processing VU (706.230) - Andi Rexha 02/04/2020 Word Embeddings

Word Embeddings Natural Language Processing VU (706.230) - Andi Rexha 02/04/2020 Word Embeddings

Word Embeddings Natural Language Processing VU (706.230) - Andi Rexha 02/04/2020 Word Embeddings Agenda Traditional NLP Word Embeddings-1 Word Embeddings-2 Text preprocessing Topic Modeling ELMo Bag-of-words model Neural

900 views • 64 slides
Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word embeddings: Early work Word embeddings via language models Word2vec and Glove Evaluating embeddings Design choices and open questions 1

1.25k views • 75 slides
Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word embeddings: Early work Word embeddings via language models Word2vec and Glove Evaluating embeddings Design choices and open questions 1

828 views • 15 slides
Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word embeddings: Early work Word embeddings via language models Word2vec and Glove Evaluating embeddings Design choices and open questions 1

2k views • 24 slides
Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word embeddings: Early work Word embeddings via language models Word2vec and Glove Evaluating embeddings Design choices and open questions 1

422 views • 17 slides
Inside Out: Two Jointly Predictive Models for Word Representations and Phrase Representations Fei

Inside Out: Two Jointly Predictive Models for Word Representations and Phrase Representations Fei

Inside Out: Two Jointly Predictive Models for Word Representations and Phrase Representations Fei Sun , Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng ofey.sunfei@gmail.com CAS Key Lab of Network Data Science and Technology Institute of

701 views • 37 slides
Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky & Martin How to

Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky & Martin How to

Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky & Martin How to generate vector embeddings? One approach: feedforward neural language models Training a neural language model just to get word embeddings is expensive!

569 views • 38 slides
Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky & Martin How to

Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky & Martin How to

Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky & Martin How to generate vector embeddings? One approach: feedforward neural language models Training a neural language model just to get word embeddings is expensive!

537 views • 35 slides
Learning Word Embeddings for Low-resource Languages by PU Learning Chao Jiang, Hsiang-Fu Yu,

Learning Word Embeddings for Low-resource Languages by PU Learning Chao Jiang, Hsiang-Fu Yu,

Learning Word Embeddings for Low-resource Languages by PU Learning Chao Jiang, Hsiang-Fu Yu, Cho-Jui Hsieh, Kai-Wei Chang 1 Word Embeddings are useful Many successful stories Named entity recognition Document ranking

697 views • 28 slides
IN5550: Neural Methods in Natural Language Processing Lecture 6 Evaluating Word Embeddings and

IN5550: Neural Methods in Natural Language Processing Lecture 6 Evaluating Word Embeddings and

IN5550: Neural Methods in Natural Language Processing Lecture 6 Evaluating Word Embeddings and Using them in Deep Neural Networks Andrey Kutuzov, Vinit Ravishankar, Lilja vrelid, Stephan Oepen, & Erik Velldal University of Oslo 21

581 views • 42 slides
Mixed membership word embeddings: Corpus-specific embeddings without big data James Foulds

Mixed membership word embeddings: Corpus-specific embeddings without big data James Foulds

Mixed membership word embeddings: Corpus-specific embeddings without big data James Foulds University of California, San Diego Southern California Machine Learning Symposium, Caltech, 11/18/2018 Word Embeddings Language models which learn

537 views • 32 slides
Word Embeddings Revisited: Contextual Embeddings CS 6956: Deep Learning for NLP Overview

Word Embeddings Revisited: Contextual Embeddings CS 6956: Deep Learning for NLP Overview

Word Embeddings Revisited: Contextual Embeddings CS 6956: Deep Learning for NLP Overview Word types and tokens Training contextual embeddings Embeddings from Language Models (ELMo) 1 Overview Word types and tokens Training

670 views • 40 slides
Sparse Coding of Neural Word Embeddings for Multilingual Sequence Labeling Gbor Berend

Sparse Coding of Neural Word Embeddings for Multilingual Sequence Labeling Gbor Berend

Sparse Coding of Neural Word Embeddings for Multilingual Sequence Labeling Gbor Berend 31/07/2017 Vancouver, ACL Continuous word representations apple [1 0 0 0 0 0 0 0 0 0] [3.2 -1.5] ... banana [0 0 0 0 1 0 0

702 views • 30 slides
SI425 : NLP Set 14 Neural NLP Fall 2020 : Chambers Why are these so different? Last time :

SI425 : NLP Set 14 Neural NLP Fall 2020 : Chambers Why are these so different? Last time :

SI425 : NLP Set 14 Neural NLP Fall 2020 : Chambers Why are these so different? Last time : Word2Vec learned word embeddings This time : use word embeddings as input to classifiers Recall: Logistic Regression x = it was the best of times it

740 views • 44 slides
Word embeddings Rappel Embeddings ( pas Word Embeddings ) Est une lookup table Formalisme:

Word embeddings Rappel Embeddings ( pas Word Embeddings ) Est une lookup table Formalisme:

Word embeddings Rappel Embeddings ( pas Word Embeddings ) Est une lookup table Formalisme: Index dun mot: w i Table dembeddings (lookup matrix): V Embedding: e i e i = V( w i ) Reprsentation dun mot

780 views • 47 slides
Natural Language Processing with Deep Learning Word Embeddings Navid Rekab-Saz

Natural Language Processing with Deep Learning Word Embeddings Navid Rekab-Saz

Natural Language Processing with Deep Learning Word Embeddings Navid Rekab-Saz navid.rekabsaz@jku.at Institute of Computational Perception Agenda Introduction Count-based word representation Prediction-based word embedding

633 views • 48 slides
IN5550: Neural Methods in Natural Language Processing Lecture 11/1 Contextualized embeddings

IN5550: Neural Methods in Natural Language Processing Lecture 11/1 Contextualized embeddings

IN5550: Neural Methods in Natural Language Processing Lecture 11/1 Contextualized embeddings Andrey Kutuzov University of Oslo 14 April 2020 1 Contents Brief Recap 1 Problems of static word embeddings 2 Solution: contextualized

372 views • 24 slides
Lecture 26 Word Embeddings and Recurrent Nets Julia Hockenmaier juliahmr@illinois.edu 3324

Lecture 26 Word Embeddings and Recurrent Nets Julia Hockenmaier juliahmr@illinois.edu 3324

CS447: Natural Language Processing http://courses.engr.illinois.edu/cs447 Lecture 26 Word Embeddings and Recurrent Nets Julia Hockenmaier juliahmr@illinois.edu 3324 Siebel Center Where were at Lecture 25: Word Embeddings and neural LMs

767 views • 49 slides
Learning Word Embeddings from Speech NIPS Workshop on Machine Learning for Audio Signal Processing

Learning Word Embeddings from Speech NIPS Workshop on Machine Learning for Audio Signal Processing

Learning Word Embeddings from Speech NIPS Workshop on Machine Learning for Audio Signal Processing December 8, 2017 Yu-An Chung James Glass MIT Computer Science and Artificial Intelligence Laboratory Cambridge, MA Outline Motivation

1.29k views • 19 slides
Lecture 7: Word Embeddings Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse

Lecture 7: Word Embeddings Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse

Lecture 7: Word Embeddings Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse webpage: http://kwchang.net/teaching/NLP16 6501 Natural Language Processing 1 This lecture v Learning word vectors (Cont.) v Representation learning in

1.15k views • 33 slides
Recurrent Neural Networks (RNNs) for NLP MACHINE LEARNING MEETUP DR. ANA PELETEIRO RAMALLO

Recurrent Neural Networks (RNNs) for NLP MACHINE LEARNING MEETUP DR. ANA PELETEIRO RAMALLO

Recurrent Neural Networks (RNNs) for NLP MACHINE LEARNING MEETUP DR. ANA PELETEIRO RAMALLO 29-05-2017 TABLE OF CONTENTS DEEP LEARNING FOR NLP WORD EMBEDDINGS RECURRENT NEURAL NETWORKS (RNNs) APPLICATION UPSKILLING 2 ZALANDO Our

712 views • 23 slides
Sparse Word Embeddings Using 1 Regularized Online Learning Fei Sun , Jiafeng Guo, Yanyan Lan,

Sparse Word Embeddings Using 1 Regularized Online Learning Fei Sun , Jiafeng Guo, Yanyan Lan,

Sparse Word Embeddings Using 1 Regularized Online Learning Fei Sun , Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng July 14, 2016 ofey.sunfei@gmail.com, { guojiafeng, lanyanyan,junxu, cxq } @ict.ac.cn CAS Key Lab of Network Data Science and

757 views • 36 slides
Jointly Learning Word Representations and Composition Functions Using Predicate-Argument

Jointly Learning Word Representations and Composition Functions Using Predicate-Argument

Jointly Learning Word Representations and Composition Functions Using Predicate-Argument Structures Kazuma Hashimoto (UT) Pontus Stenetorp (UT) Makoto Miwa (TTI) Yoshimasa Tsuruoka (UT) U niversity of T okyo ( UT ) T oyota T echnological I

1.39k views • 119 slides
Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun ,

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun ,

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun , Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng ofey.sunfei@gmail.com CAS Key Lab of Network Data Science and Technology Institute of Computing

1.02k views • 41 slides

More recommend