word embeddings
play

Word Embeddings Luke Zettlemoyer (Slides adapted from Danqi Chen, - PowerPoint PPT Presentation

Dec 08, 2023 •413 likes •787 views

CSEP 517 Natural Language Processing Word Embeddings Luke Zettlemoyer (Slides adapted from Danqi Chen, Greg Durrett, Chris Manning, Dan Jurafsky) How to represent words? N-gram language models P ( w it is 76 F and ) It is 76 F and ___.

  1. CSEP 517 Natural Language Processing Word Embeddings Luke Zettlemoyer (Slides adapted from Danqi Chen, Greg Durrett, Chris Manning, Dan Jurafsky)

  2. How to represent words? N-gram language models P ( w ∣ it is 76 F and ) It is 76 F and ___. [0.0001, 0.1, 0, 0, 0.002, …, 0.3, …, 0] red sunny Text classification P ( y = 1 ∣ x ) = σ ( θ ⊺ w + b ) I like this movie. 👎 [0, 1, 0, 0, 0, …, 1, …, 1] w (1) [0, 1, 0, 1, 0, …, 1, …, 1] I don’t like this movie. 👏 w (2) don’t

  3. Representing words as discrete symbols In traditional NLP, we regard words as discrete symbols: hotel, conference, motel — a localist representation one 1, the rest 0’s Words can be represented by one-hot vectors: hotel = [0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0] motel = [0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0] Vector dimension = number of words in vocabulary (e.g., 500,000) Challenge: How to compute similarity of two words?

  4. Representing words by their context Distributional hypothesis : words that occur in similar contexts tend to have similar meanings J.R.Firth 1957 • “You shall know a word by the company it keeps” • One of the most successful ideas of modern statistical NLP! These context words will represent banking.

  5. Distributional hypothesis C1: A bottle of ___ is on the table. “tejuino” C2: Everybody likes ___. C3: Don’t have ___ before you drive. C4: We make ___ out of corn.

  6. Distributional hypothesis C1 C2 C3 C4 tejuino 1 1 1 1 C1: A bottle of ___ is on the table. 0 0 0 0 loud C2: Everybody likes ___. 1 0 0 0 motor-oil C3: Don’t have ___ before you drive. 0 1 0 1 tortillas 0 1 0 0 choices C4: We make ___ out of corn. 1 1 1 0 wine “words that occur in similar contexts tend to have similar meanings”

  7. Words as vectors • We’ll build a new model of meaning focusing on similarity • Each word is a vector • Similar words are “nearby in space” • A first solution: we can just use context vectors to represent the meaning of words! • word-word co-occurrence matrix:

  8. <latexit sha1_base64="Ex14HCNmlweatqm/BdVuIyGZI=">ACX3icbZFLSwMxFIUz47tWHXUlboJFqCBlpgi6EUQ3LivYKnTaIZNmajDzMI+BEvIn3Qlu/Cdm2graeiFwON+9JPckLhgV0vc/HdldW19Y3Ortl3f2d3z9g96Ilcky7OWc6fYyQIoxnpSioZeS4QWnMyFP8elfxp5JwQfPsU4KMkjROKMJxUhaK/JKnItmCL5EidamXP4o0tzBq9hmHCEdShUGml6HZih7hmtIgrLiBpjwRuXS3jYruC/rJwxE3kNv+VPCy6LYC4aYF6dyHsPRzlWKckZkiIfuAXcqARlxQzYmqhEqRA+BWNSd/KDKVEDPQ0HwNPrTOCSc7tyScur8nNEqFmKSx7ay2F4usMv9jfSWTq4GmWaEkyfDsokQxKHNYhQ1HlBMs2cQKhDm1b4X4BdlIpf2Smg0hWFx5WfTarcBvBQ8XjZvbeRyb4BicgCYIwCW4AfegA7oAg0/HdbaduvPlbri7rjdrdZ35zCH4U+7RNyRuQc=</latexit> <latexit sha1_base64="Ex14HCNmlweatqm/BdVuIyGZI=">ACX3icbZFLSwMxFIUz47tWHXUlboJFqCBlpgi6EUQ3LivYKnTaIZNmajDzMI+BEvIn3Qlu/Cdm2graeiFwON+9JPckLhgV0vc/HdldW19Y3Ortl3f2d3z9g96Ilcky7OWc6fYyQIoxnpSioZeS4QWnMyFP8elfxp5JwQfPsU4KMkjROKMJxUhaK/JKnItmCL5EidamXP4o0tzBq9hmHCEdShUGml6HZih7hmtIgrLiBpjwRuXS3jYruC/rJwxE3kNv+VPCy6LYC4aYF6dyHsPRzlWKckZkiIfuAXcqARlxQzYmqhEqRA+BWNSd/KDKVEDPQ0HwNPrTOCSc7tyScur8nNEqFmKSx7ay2F4usMv9jfSWTq4GmWaEkyfDsokQxKHNYhQ1HlBMs2cQKhDm1b4X4BdlIpf2Smg0hWFx5WfTarcBvBQ8XjZvbeRyb4BicgCYIwCW4AfegA7oAg0/HdbaduvPlbri7rjdrdZ35zCH4U+7RNyRuQc=</latexit> <latexit sha1_base64="Ex14HCNmlweatqm/BdVuIyGZI=">ACX3icbZFLSwMxFIUz47tWHXUlboJFqCBlpgi6EUQ3LivYKnTaIZNmajDzMI+BEvIn3Qlu/Cdm2graeiFwON+9JPckLhgV0vc/HdldW19Y3Ortl3f2d3z9g96Ilcky7OWc6fYyQIoxnpSioZeS4QWnMyFP8elfxp5JwQfPsU4KMkjROKMJxUhaK/JKnItmCL5EidamXP4o0tzBq9hmHCEdShUGml6HZih7hmtIgrLiBpjwRuXS3jYruC/rJwxE3kNv+VPCy6LYC4aYF6dyHsPRzlWKckZkiIfuAXcqARlxQzYmqhEqRA+BWNSd/KDKVEDPQ0HwNPrTOCSc7tyScur8nNEqFmKSx7ay2F4usMv9jfSWTq4GmWaEkyfDsokQxKHNYhQ1HlBMs2cQKhDm1b4X4BdlIpf2Smg0hWFx5WfTarcBvBQ8XjZvbeRyb4BicgCYIwCW4AfegA7oAg0/HdbaduvPlbri7rjdrdZ35zCH4U+7RNyRuQc=</latexit> <latexit sha1_base64="Ex14HCNmlweatqm/BdVuIyGZI=">ACX3icbZFLSwMxFIUz47tWHXUlboJFqCBlpgi6EUQ3LivYKnTaIZNmajDzMI+BEvIn3Qlu/Cdm2graeiFwON+9JPckLhgV0vc/HdldW19Y3Ortl3f2d3z9g96Ilcky7OWc6fYyQIoxnpSioZeS4QWnMyFP8elfxp5JwQfPsU4KMkjROKMJxUhaK/JKnItmCL5EidamXP4o0tzBq9hmHCEdShUGml6HZih7hmtIgrLiBpjwRuXS3jYruC/rJwxE3kNv+VPCy6LYC4aYF6dyHsPRzlWKckZkiIfuAXcqARlxQzYmqhEqRA+BWNSd/KDKVEDPQ0HwNPrTOCSc7tyScur8nNEqFmKSx7ay2F4usMv9jfSWTq4GmWaEkyfDsokQxKHNYhQ1HlBMs2cQKhDm1b4X4BdlIpf2Smg0hWFx5WfTarcBvBQ8XjZvbeRyb4BicgCYIwCW4AfegA7oAg0/HdbaduvPlbri7rjdrdZ35zCH4U+7RNyRuQc=</latexit> <latexit sha1_base64="LwUwX9BK80Y7mx5ulvUsqXb08A0=">ACSHicbVDLSsNAFJ3UV62vqEs3g0WoICURQTdC0Y3LCvYBTQiT6aQdOsmEmUmhpPk8Ny7d+Q1uXCjizklbqbYeGDj3nHu5d4fMyqVZb0YhZXVtfWN4mZpa3tnd8/cP2hKnghMGpgzLto+koTRiDQUVYy0Y0FQ6DPS8ge3ud8aEiEpjx7UKCZuiHoRDShGSkue6WEuK06IVN8P0iQ7gz98mJ3Ca+gEAuF07kMHd7n61ZSlznhuO2M4L4e6zDyzbFWtCeAysWekDGaoe+az0+U4CUmkMENSdmwrVm6KhKYkazkJLECA9Qj3Q0jVBIpJtOgsjgiVa6MOBCv0jBifp7IkWhlKPQ1535kXLRy8X/vE6igis3pVGcKBLh6aIgYVBxmKcKu1QrNhIE4QF1bdC3Ec6O6WzL+kQ7MUvL5PmedW2qvb9Rbl2M4ujCI7AMagAG1yCGrgDdAGDyCV/AOPown4834NL6mrQVjNnMI/qBQ+AbQvrTU</latexit> <latexit sha1_base64="LwUwX9BK80Y7mx5ulvUsqXb08A0=">ACSHicbVDLSsNAFJ3UV62vqEs3g0WoICURQTdC0Y3LCvYBTQiT6aQdOsmEmUmhpPk8Ny7d+Q1uXCjizklbqbYeGDj3nHu5d4fMyqVZb0YhZXVtfWN4mZpa3tnd8/cP2hKnghMGpgzLto+koTRiDQUVYy0Y0FQ6DPS8ge3ud8aEiEpjx7UKCZuiHoRDShGSkue6WEuK06IVN8P0iQ7gz98mJ3Ca+gEAuF07kMHd7n61ZSlznhuO2M4L4e6zDyzbFWtCeAysWekDGaoe+az0+U4CUmkMENSdmwrVm6KhKYkazkJLECA9Qj3Q0jVBIpJtOgsjgiVa6MOBCv0jBifp7IkWhlKPQ1535kXLRy8X/vE6igis3pVGcKBLh6aIgYVBxmKcKu1QrNhIE4QF1bdC3Ec6O6WzL+kQ7MUvL5PmedW2qvb9Rbl2M4ujCI7AMagAG1yCGrgDdAGDyCV/AOPown4834NL6mrQVjNnMI/qBQ+AbQvrTU</latexit> <latexit sha1_base64="LwUwX9BK80Y7mx5ulvUsqXb08A0=">ACSHicbVDLSsNAFJ3UV62vqEs3g0WoICURQTdC0Y3LCvYBTQiT6aQdOsmEmUmhpPk8Ny7d+Q1uXCjizklbqbYeGDj3nHu5d4fMyqVZb0YhZXVtfWN4mZpa3tnd8/cP2hKnghMGpgzLto+koTRiDQUVYy0Y0FQ6DPS8ge3ud8aEiEpjx7UKCZuiHoRDShGSkue6WEuK06IVN8P0iQ7gz98mJ3Ca+gEAuF07kMHd7n61ZSlznhuO2M4L4e6zDyzbFWtCeAysWekDGaoe+az0+U4CUmkMENSdmwrVm6KhKYkazkJLECA9Qj3Q0jVBIpJtOgsjgiVa6MOBCv0jBifp7IkWhlKPQ1535kXLRy8X/vE6igis3pVGcKBLh6aIgYVBxmKcKu1QrNhIE4QF1bdC3Ec6O6WzL+kQ7MUvL5PmedW2qvb9Rbl2M4ujCI7AMagAG1yCGrgDdAGDyCV/AOPown4834NL6mrQVjNnMI/qBQ+AbQvrTU</latexit> <latexit sha1_base64="LwUwX9BK80Y7mx5ulvUsqXb08A0=">ACSHicbVDLSsNAFJ3UV62vqEs3g0WoICURQTdC0Y3LCvYBTQiT6aQdOsmEmUmhpPk8Ny7d+Q1uXCjizklbqbYeGDj3nHu5d4fMyqVZb0YhZXVtfWN4mZpa3tnd8/cP2hKnghMGpgzLto+koTRiDQUVYy0Y0FQ6DPS8ge3ud8aEiEpjx7UKCZuiHoRDShGSkue6WEuK06IVN8P0iQ7gz98mJ3Ca+gEAuF07kMHd7n61ZSlznhuO2M4L4e6zDyzbFWtCeAysWekDGaoe+az0+U4CUmkMENSdmwrVm6KhKYkazkJLECA9Qj3Q0jVBIpJtOgsjgiVa6MOBCv0jBifp7IkWhlKPQ1535kXLRy8X/vE6igis3pVGcKBLh6aIgYVBxmKcKu1QrNhIE4QF1bdC3Ec6O6WzL+kQ7MUvL5PmedW2qvb9Rbl2M4ujCI7AMagAG1yCGrgDdAGDyCV/AOPown4834NL6mrQVjNnMI/qBQ+AbQvrTU</latexit> Words as vectors u · v cos ( u , v ) = k u kk v k P V i =1 u i v i cos ( u , v ) = qP V qP V i =1 u 2 i =1 v 2 i i What is the range of ? cos( ⋅ )

  9. Words as vectors Problem: not all counts are equal, words can randomly co-occur • Solution: re-weight by how likely it is for the two words to co-occur by simple chance • PPMI = Positive Pointwise Mutual Information

  10. Sparse vs dense vectors • Still, the vectors we get from word-word occurrence matrix are sparse (most are 0’s) & long (vocabulary size) • Alternative: we want to represent words as short (50-300 dimensional) & dense (real-valued) vectors • The focus of this lecture • The basis of all the modern NLP systems

Recommend

Word Embeddings Natural Language Processing VU (706.230) - Andi Rexha 02/04/2020 Word Embeddings

Word Embeddings Natural Language Processing VU (706.230) - Andi Rexha 02/04/2020 Word Embeddings

Word Embeddings Natural Language Processing VU (706.230) - Andi Rexha 02/04/2020 Word Embeddings Agenda Traditional NLP Word Embeddings-1 Word Embeddings-2 Text preprocessing Topic Modeling ELMo Bag-of-words model Neural

901 views • 64 slides
Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word embeddings: Early work Word embeddings via language models Word2vec and Glove Evaluating embeddings Design choices and open questions 1

2k views • 24 slides
Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word embeddings: Early work Word embeddings via language models Word2vec and Glove Evaluating embeddings Design choices and open questions 1

422 views • 17 slides
Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word embeddings: Early work Word embeddings via language models Word2vec and Glove Evaluating embeddings Design choices and open questions 1

828 views • 15 slides
Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word embeddings: Early work Word embeddings via language models Word2vec and Glove Evaluating embeddings Design choices and open questions 1

1.25k views • 75 slides
Word embeddings Rappel Embeddings ( pas Word Embeddings ) Est une lookup table Formalisme:

Word embeddings Rappel Embeddings ( pas Word Embeddings ) Est une lookup table Formalisme:

Word embeddings Rappel Embeddings ( pas Word Embeddings ) Est une lookup table Formalisme: Index dun mot: w i Table dembeddings (lookup matrix): V Embedding: e i e i = V( w i ) Reprsentation dun mot

780 views • 47 slides
Mixed membership word embeddings: Corpus-specific embeddings without big data James Foulds

Mixed membership word embeddings: Corpus-specific embeddings without big data James Foulds

Mixed membership word embeddings: Corpus-specific embeddings without big data James Foulds University of California, San Diego Southern California Machine Learning Symposium, Caltech, 11/18/2018 Word Embeddings Language models which learn

537 views • 32 slides
Word Embeddings through Hellinger PCA Rmi Lebret and Ronan Collobert Idiap Research Institute /

Word Embeddings through Hellinger PCA Rmi Lebret and Ronan Collobert Idiap Research Institute /

Word Embeddings through Hellinger PCA Rmi Lebret and Ronan Collobert Idiap Research Institute / EPFL EACL, 29 April 2014 Word Embeddings Continuous vector-space models. Represent word meanings with vectors capturing semantic +

831 views • 46 slides
Word Embeddings Tutorial HILA GONEN PHD STUDENT AT YOAV GOLDBERGS LAB BAR ILAN UNIVERSITY

Word Embeddings Tutorial HILA GONEN PHD STUDENT AT YOAV GOLDBERGS LAB BAR ILAN UNIVERSITY

Word Embeddings Tutorial HILA GONEN PHD STUDENT AT YOAV GOLDBERGS LAB BAR ILAN UNIVERSITY 5/3/18 Outline NLP Intro Word representations and word embeddings Word2vec models Visualizing word embeddings Word2vec in

1.18k views • 63 slides
Word Embeddings Revisited: Contextual Embeddings CS 6956: Deep Learning for NLP Overview

Word Embeddings Revisited: Contextual Embeddings CS 6956: Deep Learning for NLP Overview

Word Embeddings Revisited: Contextual Embeddings CS 6956: Deep Learning for NLP Overview Word types and tokens Training contextual embeddings Embeddings from Language Models (ELMo) 1 Overview Word types and tokens Training

670 views • 40 slides
Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction Roy Schwartz + ,

Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction Roy Schwartz + ,

Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction Roy Schwartz + , Roi Reichart * and Ari Rappoport + + The Hebrew University, * Technion IIT CoNLL 2015 Symmetric Pattern Based Word Embeddings for 2 Improved Word

917 views • 49 slides
Searching for the X-Factor: Exploring Corpus Subjectivity for Word Embeddings Maksim Tkachenko

Searching for the X-Factor: Exploring Corpus Subjectivity for Word Embeddings Maksim Tkachenko

Searching for the X-Factor: Exploring Corpus Subjectivity for Word Embeddings Maksim Tkachenko and Chong Cher Chia and Hady W. Lauw Singapore Management University Word Embeddings Dense vectors of words Unsupervised training: GloVe,

522 views • 19 slides
Pattern-based Solutions to Limitations of Leading Word Embeddings Roy Schwartz University of

Pattern-based Solutions to Limitations of Leading Word Embeddings Roy Schwartz University of

Pattern-based Solutions to Limitations of Leading Word Embeddings Roy Schwartz University of Washington NLP Seminar, February 8 th , 2016 Joint work with Roi Reichart and Ari Rappoport Background Word embeddings are great! Problem

872 views • 84 slides
Word, Sense and Contextualized Embeddings: Vector Representations of Meaning in NLP Jose

Word, Sense and Contextualized Embeddings: Vector Representations of Meaning in NLP Jose

Word, Sense and Contextualized Embeddings: Vector Representations of Meaning in NLP Jose Camacho-Collados Cardiff University, 18 March 2019 1 Outline Background Vector Space Models (word embeddings) Lexical resources Sense

1.33k views • 98 slides
Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin How to

Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin How to

Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin How to generate vector embeddings? One approach: feedforward neural language models Training a neural language model just to get word embeddings is expensive!

537 views • 35 slides
Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin How to

Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin How to

Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin How to generate vector embeddings? One approach: feedforward neural language models Training a neural language model just to get word embeddings is expensive!

569 views • 38 slides
Learning Word Embeddings for Low-resource Languages by PU Learning Chao Jiang, Hsiang-Fu Yu,

Learning Word Embeddings for Low-resource Languages by PU Learning Chao Jiang, Hsiang-Fu Yu,

Learning Word Embeddings for Low-resource Languages by PU Learning Chao Jiang, Hsiang-Fu Yu, Cho-Jui Hsieh, Kai-Wei Chang 1 Word Embeddings are useful Many successful stories Named entity recognition Document ranking

697 views • 28 slides
Humor in Word Embeddings: Cockamamie Gobbledegook for Nincompoops Limor Gultchin, Genevieve

Humor in Word Embeddings: Cockamamie Gobbledegook for Nincompoops Limor Gultchin, Genevieve

Humor in Word Embeddings: Cockamamie Gobbledegook for Nincompoops Limor Gultchin, Genevieve Patterson, Nancy Baym, Nathaniel Swinger, and Adam Tauman Kalai https://github.com/limorigu/Cockamamie-Gobbledegook Toolkit: Word embeddings

325 views • 7 slides
Lecture 8: NLP and Word Embeddings Alireza Akhavan Pour CLASS.VISION

Lecture 8: NLP and Word Embeddings Alireza Akhavan Pour CLASS.VISION

Lecture 8: NLP and Word Embeddings Alireza Akhavan Pour CLASS.VISION SRTTU A.Akhavan 1 Lecture 8 - NLP and Word Embeddings Word representation V = [a, aaron, ..., zulu, &lt;UNK&gt;] = ,

449 views • 16 slides
Motivation Current Scenario : Rising interest in vector space word embeddings and their use, given

Motivation Current Scenario : Rising interest in vector space word embeddings and their use, given

IMPROVING WORD EMBEDDINGS USING MULTIPLE WORD PROTOTYPES CS671A Course Project : Under Prof. Amitabha Mukerjee Anurendra Kumar Nishant Rai 15th October Indian Institute of Technology Kanpur Motivation Current Scenario : Rising interest in

505 views • 16 slides
USING PSEUDO-SENSES FOR IMPROVING THE EXTRACTION OF SYNONYMS FROM WORD EMBEDDINGS Olivier Ferret

USING PSEUDO-SENSES FOR IMPROVING THE EXTRACTION OF SYNONYMS FROM WORD EMBEDDINGS Olivier Ferret

USING PSEUDO-SENSES FOR IMPROVING THE EXTRACTION OF SYNONYMS FROM WORD EMBEDDINGS Olivier Ferret CONTEXT AND OBJECTIVES Context semantic specialization of word embeddings most approaches following Retrofitting [Faruqui et al.,

341 views • 11 slides
Generalizing Word Embeddings using Bag of Subwords Jinman Zhao , Sidharth Mudgal, Yingyu Liang

Generalizing Word Embeddings using Bag of Subwords Jinman Zhao , Sidharth Mudgal, Yingyu Liang

Generalizing Word Embeddings using Bag of Subwords Jinman Zhao , Sidharth Mudgal, Yingyu Liang University of Wisconsin-Madison Nov. 2, 2018 @ EMNLP Word Embeddings the [ -0.1 0.1 0.3 ... ] Belgium officially the Kingdom of Belgium, is a

244 views • 22 slides
Deep convolutional acoustic word embeddings using word-pair side information Herman Kamper 1 ,

Deep convolutional acoustic word embeddings using word-pair side information Herman Kamper 1 ,

Deep convolutional acoustic word embeddings using word-pair side information Herman Kamper 1 , Weiran Wang 2 , Karen Livescu 2 1 CSTR and ILCC, School of Informatics, University of Edinburgh, UK 2 Toyota Technological Institute at Chicago, USA

1.03k views • 76 slides
word2vec Kuan-Ting Lai 2020/5/28 Word2vec (Word Embeddings) Embed one-hot encoded word

word2vec Kuan-Ting Lai 2020/5/28 Word2vec (Word Embeddings) Embed one-hot encoded word

word2vec Kuan-Ting Lai 2020/5/28 Word2vec (Word Embeddings) Embed one-hot encoded word vectors into dense vectors Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. &quot;Distributed representations of words and

529 views • 38 slides

More recommend