unsupervised language learning representation learning
play

Unsupervised Language Learning: Representation Learning for NLP - PowerPoint PPT Presentation

Mar 02, 2024 •127 likes •898 views

Unsupervised Language Learning: Representation Learning for NLP Unsupervised Language Learning: Representation Learning for NLP Katia Shutova ILLC University of Amsterdam 3 April 2018 Unsupervised Language Learning: Representation Learning

  1. Unsupervised Language Learning: Representation Learning for NLP Unsupervised Language Learning: Representation Learning for NLP Katia Shutova ILLC University of Amsterdam 3 April 2018

  2. Unsupervised Language Learning: Representation Learning for NLP Taught by... ◮ Lecturers: Katia Shutova and Wilker Aziz ◮ Teaching assistant: Samira Abnar

  3. Unsupervised Language Learning: Representation Learning for NLP Lecture 1: Introduction Overview of the course Distributional semantics Count-based models Similarity Distributional word clustering

  4. Unsupervised Language Learning: Representation Learning for NLP Overview of the course Overview of the course ◮ This course is about learning meaning representations ◮ Methods for learning meaning representations from linguistic data ◮ Analysis of meaning representations learnt ◮ Applications ◮ This is a research seminar ◮ Lectures ◮ You will present and critique research papers, ◮ implement and evaluate representation learning methods ◮ and analyse their behaviour

  5. Unsupervised Language Learning: Representation Learning for NLP Overview of the course Overview of the course We will cover the following topics: ◮ Introduction to distributional semantics ◮ Learning word and phrase representations – deep learning ◮ Learning word representations – Bayesian learning ◮ Multilingual word representations ◮ Multimodal word representations (language and vision) ◮ Applications: NLP and neuroscience

  6. Unsupervised Language Learning: Representation Learning for NLP Overview of the course Assessment Work in groups of 2. ◮ Presentation and participation (20%) ◮ Present 1 paper per group in class ◮ Practical assignments, assessed by reports 1. Analysis of the properties of word representations (10%) 2. Implement 3 representation learning methods (20%) 3. Evaluate in the context of external NLP applications – final report (50%) More information at the first lab session on Thursday, 5 April.

  7. Unsupervised Language Learning: Representation Learning for NLP Overview of the course Also note: Course materials and more info: https://uva-slpl.github.io/ull/ Contact ◮ Main contact – Samira: s.abnar@uva.nl ◮ Katia: e.shutova@uva.nl ◮ Wilker: w.aziz@uva.nl Email Samira by Thursday, 5 April with details of your group. ◮ names of the students ◮ their email addresses ◮ subject: ULL group assignment

  8. Unsupervised Language Learning: Representation Learning for NLP Overview of the course Natural Language Processing Many popular applications ◮ Information retrieval ◮ Machine translation ◮ Question answering ◮ Dialogue systems ◮ Sentiment analysis ◮ Recently: fact checking etc.

  9. Unsupervised Language Learning: Representation Learning for NLP Overview of the course Why is NLP difficult? Similar strings mean different things, different strings mean the same thing. ◮ Synonymy: different strings can mean the same thing The King’s speech gave the much needed reassurance to his people. His majesty’s address reassured the crowds. ◮ Ambiguity: same strings can mean different things His majesty’s address reassured the crowds. His majesty’s address is Buckingham Palace, London SW1A 1AA.

  10. Unsupervised Language Learning: Representation Learning for NLP Overview of the course Why is NLP difficult? Similar strings mean different things, different strings mean the same thing. ◮ Synonymy: different strings can mean the same thing The King’s speech gave the much needed reassurance to his people. His majesty’s address reassured the crowds. ◮ Ambiguity: same strings can mean different things His majesty’s address reassured the crowds. His majesty’s address is Buckingham Palace, London SW1A 1AA.

  11. Unsupervised Language Learning: Representation Learning for NLP Overview of the course Why is NLP difficult? Similar strings mean different things, different strings mean the same thing. ◮ Synonymy: different strings can mean the same thing The King’s speech gave the much needed reassurance to his people. His majesty’s address reassured the crowds. ◮ Ambiguity: same strings can mean different things His majesty’s address reassured the crowds. His majesty’s address is Buckingham Palace, London SW1A 1AA.

  12. Unsupervised Language Learning: Representation Learning for NLP Overview of the course Why is NLP difficult? Similar strings mean different things, different strings mean the same thing. ◮ Synonymy: different strings can mean the same thing The King’s speech gave the much needed reassurance to his people. His majesty’s address reassured the crowds. ◮ Ambiguity: same strings can mean different things His majesty’s address reassured the crowds. His majesty’s address is Buckingham Palace, London SW1A 1AA.

  13. Unsupervised Language Learning: Representation Learning for NLP Overview of the course Why is NLP difficult? Similar strings mean different things, different strings mean the same thing. ◮ Synonymy: different strings can mean the same thing The King’s speech gave the much needed reassurance to his people. His majesty’s address reassured the crowds. ◮ Ambiguity: same strings can mean different things His majesty’s address reassured the crowds. His majesty’s address is Buckingham Palace, London SW1A 1AA.

  14. Unsupervised Language Learning: Representation Learning for NLP Overview of the course Wouldn’t it be better if . . . ? The properties which make natural language difficult to process are essential to human communication: ◮ Flexible ◮ Learnable, but expressive and compact ◮ Emergent, evolving systems Synonymy and ambiguity go along with these properties. Natural language communication can be indefinitely precise: ◮ Ambiguity is mostly local (for humans) ◮ resolved by immediate context ◮ but requires world knowledge

  15. Unsupervised Language Learning: Representation Learning for NLP Overview of the course Wouldn’t it be better if . . . ? The properties which make natural language difficult to process are essential to human communication: ◮ Flexible ◮ Learnable, but expressive and compact ◮ Emergent, evolving systems Synonymy and ambiguity go along with these properties. Natural language communication can be indefinitely precise: ◮ Ambiguity is mostly local (for humans) ◮ resolved by immediate context ◮ but requires world knowledge

  16. Unsupervised Language Learning: Representation Learning for NLP Overview of the course World knowledge... ◮ Impossible to hand-code at a large-scale ◮ either limited domain applications ◮ or learn approximations from the data

  17. Unsupervised Language Learning: Representation Learning for NLP Distributional semantics Distributional hypothesis You shall know a word by the company it keeps (Firth) The meaning of a word is defined by the way it is used (Wittgenstein). it was authentic scrumpy, rather sharp and very strong we could taste a famous local product — scrumpy spending hours in the pub drinking scrumpy Cornish Scrumpy Medium Dry. £19.28 - Case

  18. Unsupervised Language Learning: Representation Learning for NLP Distributional semantics Distributional hypothesis You shall know a word by the company it keeps (Firth) The meaning of a word is defined by the way it is used (Wittgenstein). it was authentic scrumpy, rather sharp and very strong we could taste a famous local product — scrumpy spending hours in the pub drinking scrumpy Cornish Scrumpy Medium Dry. £19.28 - Case

  19. Unsupervised Language Learning: Representation Learning for NLP Distributional semantics Distributional hypothesis You shall know a word by the company it keeps (Firth) The meaning of a word is defined by the way it is used (Wittgenstein). it was authentic scrumpy, rather sharp and very strong we could taste a famous local product — scrumpy spending hours in the pub drinking scrumpy Cornish Scrumpy Medium Dry. £19.28 - Case

  20. Unsupervised Language Learning: Representation Learning for NLP Distributional semantics Distributional hypothesis You shall know a word by the company it keeps (Firth) The meaning of a word is defined by the way it is used (Wittgenstein). it was authentic scrumpy, rather sharp and very strong we could taste a famous local product — scrumpy spending hours in the pub drinking scrumpy Cornish Scrumpy Medium Dry. £19.28 - Case

  21. Unsupervised Language Learning: Representation Learning for NLP Distributional semantics Distributional hypothesis You shall know a word by the company it keeps (Firth) The meaning of a word is defined by the way it is used (Wittgenstein). it was authentic scrumpy, rather sharp and very strong we could taste a famous local product — scrumpy spending hours in the pub drinking scrumpy Cornish Scrumpy Medium Dry. £19.28 - Case

  22. Unsupervised Language Learning: Representation Learning for NLP Distributional semantics Scrumpy

  23. Unsupervised Language Learning: Representation Learning for NLP Distributional semantics Distributional hypothesis This leads to the distributional hypothesis about word meaning: ◮ the context surrounding a given word provides information about its meaning; ◮ words are similar if they share similar linguistic contexts; ◮ semantic similarity ≈ distributional similarity.

Recommend

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised learning: X - y pairs, f(x) function approximation Unsupervised learning: only X, no y Exploring the space of X measurements,

441 views • 14 slides
Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of

Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of

UNSUPERVISED LEARNING IN R Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of clustering finding groups of homogeneous items Next up, dimensionality reduction Find structure in features

904 views • 29 slides
Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation

Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation

Code & Data : github.com/snakeztc/NeuralDialog-LAED Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation Tiancheng Zhao, Kyusong Lee and Maxine Eskenazi Language Technologies Institute, Carnegie

730 views • 28 slides
Unsupervised Visual Representation Learning by Context Prediction Berkan Demirel Most slides in

Unsupervised Visual Representation Learning by Context Prediction Berkan Demirel Most slides in

Unsupervised Visual Representation Learning by Context Prediction Berkan Demirel Most slides in this representation are adopted from authors' original presentation at ICCV 2015 ImageNet + Deep Learning Beagle - Image Retrieval - Detection

2.18k views • 33 slides
Unsupervised Learning of the Morphology of a Natural Language John Goldsmith University of

Unsupervised Learning of the Morphology of a Natural Language John Goldsmith University of

Unsupervised Learning of the Morphology of a Natural Language John Goldsmith University of Chicago Abstract This study reports the results of using Minimum Description Length analysis to model unsupervised learning of the morphology of

1.15k views • 101 slides
12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from Wanli Ouyang, Zsolt Kira, Lawrence Neal, Raymond Yeh, Junting Lou and Teck-Yian Lim Unsupervised Learning in General Unsupervised learning is

1.48k views • 115 slides
Unsupervised Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Unsupervised

Unsupervised Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Unsupervised

Unsupervised Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Unsupervised Learning Unsupervised Learning Setting Supervised learning requires the availability of labelled examples Labelling examples can be an extremely

752 views • 27 slides
Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on supervised learning methods such as regression and classification. In that setting we observe both a set of features X 1 , X 2 , . . . , X p for each

934 views • 50 slides
Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on supervised learning methods such as regression and classification. In that setting we observe both a set of features X 1 , X 2 , . . . , X p for each

721 views • 60 slides
Unsupervised Learning and Clustering l In unsupervised learning you are given a data set with no

Unsupervised Learning and Clustering l In unsupervised learning you are given a data set with no

Unsupervised Learning and Clustering l In unsupervised learning you are given a data set with no output classifications (labels) l Clustering is an important type of unsupervised learning PCA was another type of unsupervised learning l The

858 views • 38 slides
Unit 2: Natural Language Learning Unsupervised Learning (EM, forward-backward, inside-outside)

Unit 2: Natural Language Learning Unsupervised Learning (EM, forward-backward, inside-outside)

Natural Language Processing Spring 2017 Unit 2: Natural Language Learning Unsupervised Learning (EM, forward-backward, inside-outside) Liang Huang liang.huang.sh@gmAYl.com Review of Noisy-Channel Model CS 562 - EM 2 Example 1:

469 views • 23 slides
Language and Computers Unsupervised Learning Features & Classifying Documents Evidence

Language and Computers Unsupervised Learning Features & Classifying Documents Evidence

Language and Computers Classifying Documents Introduction Language Identification Machine Learning Supervised Learning Language and Computers Unsupervised Learning Features & Classifying Documents Evidence Measuring sucess

800 views • 45 slides
Unsupervised Learning of Visual Representation by Solving Jigsaw Puzzles, ECCV 16 2018/11/27

Unsupervised Learning of Visual Representation by Solving Jigsaw Puzzles, ECCV 16 2018/11/27

Unsupervised Learning of Visual Representation by Solving Jigsaw Puzzles, ECCV 16 2018/11/27 20173130 Jaeyoon Kim CS688 Paper Presentation Image Retrieval with Mixed initiative and Multimodal Feedback, BMVC 18 The system based on

336 views • 18 slides
An Unsupervised Autoregressive Model for Speech Representation Learning Yu-An Chung Wei-Ning Hsu

An Unsupervised Autoregressive Model for Speech Representation Learning Yu-An Chung Wei-Ning Hsu

An Unsupervised Autoregressive Model for Speech Representation Learning Yu-An Chung Wei-Ning Hsu Hao Tang James Glass Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge, Massachusetts,

1.24k views • 13 slides
Stable and Efficient Representation Learning with Nonnegativity Constraints Tsung-Han Lin and

Stable and Efficient Representation Learning with Nonnegativity Constraints Tsung-Han Lin and

Stable and Efficient Representation Learning with Nonnegativity Constraints Tsung-Han Lin and H.T. Kung Unsupervised Representation Learning Layer 3 Representation Layer 2 Representation Encoding Sparse encoder Layer 1 Representation

304 views • 18 slides
Machine Learning for NLP Unsupervised Learning Aurlie Herbelot 2019 Centre for Mind/Brain

Machine Learning for NLP Unsupervised Learning Aurlie Herbelot 2019 Centre for Mind/Brain

Machine Learning for NLP Unsupervised Learning Aurlie Herbelot 2019 Centre for Mind/Brain Sciences University of Trento 1 Unsupervised learning In unsupervised learning, we learn without training data. The idea is to find a

998 views • 67 slides
Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning

Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning

Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning Jiasen Lu April 21, 2020 1 1 Vision and Language Visual Question Answering Image Captioning Visual Commonsense Reasoning Refer Expression 2

1.13k views • 53 slides
Representation Learning Lecture slides for Chapter 15 of Deep Learning www.deeplearningbook.org

Representation Learning Lecture slides for Chapter 15 of Deep Learning www.deeplearningbook.org

Representation Learning Lecture slides for Chapter 15 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2017-10-03 Unsupervised Pretraining Usually Hurts but Sometimes Helps Harm done by pretraining (Ma et al, 2015) Average advantage

586 views • 12 slides
Unsupervised Learning Introduction Nakul Verma Unsupervised Learning What can we learn from

Unsupervised Learning Introduction Nakul Verma Unsupervised Learning What can we learn from

Unsupervised Learning Introduction Nakul Verma Unsupervised Learning What can we learn from data when label information is not available? Supervised learning framework Supervised learning Data: Assumption: there is a (relatively simple)

442 views • 16 slides
Adversarial Fisher Vectors For Unsupervised Representation Learning Shuangfei Zhai, Walter

Adversarial Fisher Vectors For Unsupervised Representation Learning Shuangfei Zhai, Walter

Adversarial Fisher Vectors For Unsupervised Representation Learning Shuangfei Zhai, Walter Talbott, Carlos Guestrin, Joshua M. Susskind Apple Inc. Questions about GANs Is the discriminator useful at test time? Do GANs learn

397 views • 7 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

405 views • 26 slides
Unsupervised Deep Learning Tutorial Part 1 Alex Graves MarcAurelio Ranzato NeurIPS, 3

Unsupervised Deep Learning Tutorial Part 1 Alex Graves MarcAurelio Ranzato NeurIPS, 3

Unsupervised Deep Learning Tutorial Part 1 Alex Graves MarcAurelio Ranzato NeurIPS, 3 December 2018 Part 1 Alex Graves Introduction to unsupervised learning Autoregressive models Representation learning

1.47k views • 90 slides
Clustering ! Unsupervised learning: learning without a teacher ! Only features and no labels

Clustering ! Unsupervised learning: learning without a teacher ! Only features and no labels

So far in the course Supervised learning: learning with a teacher ! You had training data which was (feature, label) pairs and the goal was to learn a mapping from features to labels Clustering ! Unsupervised learning: learning without a

164 views • 12 slides
Practical Unsupervised Learning INFO/CS 4300, Spring 2016 Jack Hessel Unsupervised Learning is

Practical Unsupervised Learning INFO/CS 4300, Spring 2016 Jack Hessel Unsupervised Learning is

Practical Unsupervised Learning INFO/CS 4300, Spring 2016 Jack Hessel Unsupervised Learning is Cool! But how can we use this in our projects? 1. 2. But first, lets look at our dataset... Data Dimensionality! Probability of Lung Cancer

999 views • 55 slides

More recommend