discriminative language models
play

Discriminative Language Models Prof. Sameer Singh CS 295: - PowerPoint PPT Presentation

Sep 19, 2022 •293 likes •649 views

Discriminative Language Models Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 January 26, 2017 Based on slides from Noah Smith, Richard Socher, and everyone else they copied from. Language Models Probability of a Sentence Is a

  1. Discriminative Language Models Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 January 26, 2017 Based on slides from Noah Smith, Richard Socher, and everyone else they copied from.

  2. Language Models Probability of a Sentence • Is a given sentence something you would expect to see? • Syntactically (grammar) and Semantically (meaning) Probability of the Next Word • Predict what comes next for a given sequence of words. • Think of it as V‐way classification CS 295: STATISTICAL NLP (WINTER 2017) 2

  3. Outline Discriminative Language Models Feed‐forward Neural Networks Recurrent Neural Networks Upcoming.. CS 295: STATISTICAL NLP (WINTER 2017) 3

  4. Outline Discriminative Language Models Feed‐forward Neural Networks Recurrent Neural Networks Upcoming.. CS 295: STATISTICAL NLP (WINTER 2017) 4

  5. Logistic Regression Model CS 295: STATISTICAL NLP (WINTER 2017) 5

  6. N‐Grams as Logistic Reg. CS 295: STATISTICAL NLP (WINTER 2017) 6

  7. Other features… CS 295: STATISTICAL NLP (WINTER 2017) 7

  8. Outline Discriminative Language Models Feed‐forward Neural Networks Recurrent Neural Networks Upcoming.. CS 295: STATISTICAL NLP (WINTER 2017) 8

  9. Logistic Reg. w/ Embeddings CS 295: STATISTICAL NLP (WINTER 2017) 9

  10. Neural Networks CS 295: STATISTICAL NLP (WINTER 2017) 10

  11. Activation Functions sigmoid softmax tanh And many others… ReLUs, PReLUs, ELU, step, max, and so on.. CS 295: STATISTICAL NLP (WINTER 2017) 11

  12. Why do they work? https://colah.github.io CS 295: STATISTICAL NLP (WINTER 2017) 12

  13. Why do they work? z x2 y x1 CS 295: STATISTICAL NLP (WINTER 2017) 13

  14. Simulated Example https://github.com/clab/cnn/blob/master/examples/xor.cc CS 295: STATISTICAL NLP (WINTER 2017) 14

  15. Simple Feedforward NN LM Bigram Model CS 295: STATISTICAL NLP (WINTER 2017) 15

  16. Simple Feedforward NN LM N‐gram Model CS 295: STATISTICAL NLP (WINTER 2017) 16

  17. Deep Feedforward NN LM Bengio et al. 2003 CS 295: STATISTICAL NLP (WINTER 2017) 17

  18. Outline Discriminative Language Models Feed‐forward Neural Networks Recurrent Neural Networks Upcoming.. CS 295: STATISTICAL NLP (WINTER 2017) 18

  19. Sequence View of Simple NNs CS 295: STATISTICAL NLP (WINTER 2017) 19

  20. Recurrent Neural Networks CS 295: STATISTICAL NLP (WINTER 2017) 20

  21. Example: “I love food” love food <eos> love food I CS 295: STATISTICAL NLP (WINTER 2017) 21

  22. Power of RNNs: Characters! http://karpathy.github.io/2015/05/21/rnn‐effectiveness/ CS 295: STATISTICAL NLP (WINTER 2017) 22

  23. Char‐RNNs: Shakespeare! CS 295: STATISTICAL NLP (WINTER 2017) 23

  24. Char‐RNNs: Wikipedia! CS 295: STATISTICAL NLP (WINTER 2017) 24

  25. Char‐RNNs: Linux Code! CS 295: STATISTICAL NLP (WINTER 2017) 25

  26. Extension: Stacking CS 295: STATISTICAL NLP (WINTER 2017) 26

  27. Extension: Bidirectional RNNs CS 295: STATISTICAL NLP (WINTER 2017) 27

  28. Deep Bidirectional RNNs CS 295: STATISTICAL NLP (WINTER 2017) 28

  29. Extension: GRUs Gated Recurrent Units CS 295: STATISTICAL NLP (WINTER 2017) 29

  30. Extension: GRUs Gated Recurrent Units CS 295: STATISTICAL NLP (WINTER 2017) 30

  31. Estimating Parameters Beyond the scope of the course • Lots of tricks, heuristics, “domain knowledge” • Lot of engineering for efficiency, e.g. GPUs • New training algorithms being proposed every year • sometimes, architecture‐specific • Lots of available tools you can use! • Tensorflow, Torch, Keras, MxNET, etc. CS 295: STATISTICAL NLP (WINTER 2017) 31

  32. Outline Discriminative Language Models Feed‐forward Neural Networks Recurrent Neural Networks Upcoming.. CS 295: STATISTICAL NLP (WINTER 2017) 32

  33. Homework 1 so far… Public Private CS 295: STATISTICAL NLP (WINTER 2017) 33

  34. Ruslan Salakhutdinov Professor at Carnegie Mellon University Director of Artificial Intelligence, Apple Inc. Learning Deep Unsupervised and Multimodal Models Location : DBH 6011 Time : 11am ‐ 12pm Date: January 27, 2017 Meeting with PhD students, will post on Piazza CS 295: STATISTICAL NLP (WINTER 2017) 34

  35. Upcoming… • Homework 1 is due tonight: January 26, 2017 • Write‐up, data, and code for Homework 2 is up Homework • Homework 2 is due: February 9, 2017 • Proposal is due: February 7, 2017 (~2 weeks) Project • Only 2 pages CS 295: STATISTICAL NLP (WINTER 2017) 35

Recommend

Discriminative Models Joakim Nivre Uppsala University Department of Linguistics and Philology

Discriminative Models Joakim Nivre Uppsala University Department of Linguistics and Philology

Discriminative Models Joakim Nivre Uppsala University Department of Linguistics and Philology joakim.nivre@lingfil.uu.se Discriminative Models 1(11) 1. Generative and Discriminative Models 2. Log-Linear Models 3. Local Discriminative Models

278 views • 11 slides
MaxEnt Models and Discriminative Estimation Gerald Penn CS224N/Ling284 [based on slides by

MaxEnt Models and Discriminative Estimation Gerald Penn CS224N/Ling284 [based on slides by

MaxEnt Models and Discriminative Estimation Gerald Penn CS224N/Ling284 [based on slides by Christopher Manning and Dan Klein] Introduction So far weve looked at generative models Language models, Naive Bayes, IBM MT In

593 views • 57 slides
Structured Discriminative Models for Speech Recognition Mark Gales - work with Anton Ragni,

Structured Discriminative Models for Speech Recognition Mark Gales - work with Anton Ragni,

Structured Discriminative Models for Speech Recognition Mark Gales - work with Anton Ragni, Austin Zhang, Rogier van Dalen September 2012 Cambridge University Engineering Department Symposium on Machine Learning in Speech and Language

697 views • 54 slides
Three models for discriminative machine Three models for discriminative machine translation using

Three models for discriminative machine Three models for discriminative machine translation using

Three models for discriminative machine Three models for discriminative machine translation using Global Lexical Selection translation using Global Lexical Selection and Sentence Reconstruction and Sentence Reconstruction Sriram Venkatapathy

645 views • 32 slides
Discriminative Metric Learning in Nearest Neighbor Models for Image Annotation Matthieu

Discriminative Metric Learning in Nearest Neighbor Models for Image Annotation Matthieu

Discriminative Metric Learning in Nearest Neighbor Models for Image Annotation Matthieu Guillaumin, Thomas Mensink, Jakob Verbeek &amp; Cordelia Schmid LEAR Team, INRIA Rh one-Alpes Grenoble, France Discriminative Metric Learning in Nearest

555 views • 40 slides
Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution ? What is language? never start with a dictionary definition!! the language of Google search query completion A language is a dialect with

1.13k views • 30 slides
Models w/ Latent Random Variables Chunting Zhou, Junxian He Site

Models w/ Latent Random Variables Chunting Zhou, Junxian He Site

CS11-747 Neural Networks for NLP Models w/ Latent Random Variables Chunting Zhou, Junxian He Site https://phontron.com/class/nn4nlp2019/ Slides from Graham Neubig Discriminative vs. Generative Models Discriminative model: calculate the

932 views • 58 slides
Discriminative Bimodal Networks for Visual Localization and Detection with Natural Language

Discriminative Bimodal Networks for Visual Localization and Detection with Natural Language

Discriminative Bimodal Networks for Visual Localization and Detection with Natural Language Queries Yuting Zhang , Luyao Yuan, Yijie Guo, Zhiyuan He, I - An Huang, Honglak Lee University of Michigan, Ann Arbor Detection with natural language

336 views • 9 slides
Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin

Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin

Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin Layton &amp; Mark Gales 9 December 2005 Cambridge University Engineering Department NIPS 2005 Augmented Statistical Models: Exploiting Generative

726 views • 25 slides
Generative Models for Discriminative Problems Chris Dyer DeepMind ASRU 2017 December 19,

Generative Models for Discriminative Problems Chris Dyer DeepMind ASRU 2017 December 19,

Generative Models for Discriminative Problems Chris Dyer DeepMind ASRU 2017 December 19, 2017 Terminological clarification A discriminative problem : for some input x , find Y ( x ) the most likely y in a set A

1.71k views • 91 slides
Hallucinating system outputs for discriminative language modeling Brian Roark Center for Spoken

Hallucinating system outputs for discriminative language modeling Brian Roark Center for Spoken

Hallucinating system outputs for discriminative language modeling Brian Roark Center for Spoken Language Understanding, OHSU Joint work with D. Bikel, C. Callison-Burch, Y. Cao, A. C elebi, E. Dikici, N. Glenn, K. Hall, E. Hasler, D.

836 views • 32 slides
Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography

Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography

Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography Poster #53 Andrew C. Miller with Ziad Obermeyer, John P . Cunningham, and Sendhil Mullainathan &lt;latexit

674 views • 9 slides
4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language models: log-linear language models [9, 4], which take a very di ff erent approach than the count-based n -grams described above. 11 4.1 Model

501 views • 8 slides
Natural Language Processing Info 159/259 Lecture 3: Text classification 2 (Aug 31, 2017)

Natural Language Processing Info 159/259 Lecture 3: Text classification 2 (Aug 31, 2017)

Natural Language Processing Info 159/259 Lecture 3: Text classification 2 (Aug 31, 2017) David Bamman, UC Berkeley Generative vs. Discriminative models Generative models specify a joint distribution over the labels and the data. With

611 views • 38 slides
Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and

Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and

A New Framework For Hybrid Models An Application: Deep Hybrid Models Supervised and Semi-Supervised Experiments Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and Stefano Ermon Department of Computer

523 views • 29 slides
Generative vs. discriminative Generative Discriminative Belief network A is more More

Generative vs. discriminative Generative Discriminative Belief network A is more More

Generative vs. discriminative Generative Discriminative Belief network A is more More robust modular Dont need precise model Class-conditional densities are specification, so long as it is likely to be local, characteristic

1.09k views • 15 slides
Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models answer the question: How likely is a string of English words good English? Help with reordering p lm (the house is small) &gt; p lm (small the is

1.06k views • 51 slides
Linear Models for Classification Greg Mori - CMPT 419/726 Bishop PRML Ch. 4 Discriminant

Linear Models for Classification Greg Mori - CMPT 419/726 Bishop PRML Ch. 4 Discriminant

Discriminant Functions Generative Models Discriminative Models Linear Models for Classification Greg Mori - CMPT 419/726 Bishop PRML Ch. 4 Discriminant Functions Generative Models Discriminative Models Classification: Hand-written Digit

1.08k views • 84 slides
Linear Models for Classification Oliver Schulte - CMPT 726 Bishop PRML Ch. 4 Discriminant

Linear Models for Classification Oliver Schulte - CMPT 726 Bishop PRML Ch. 4 Discriminant

Discriminant Functions Generative Models Discriminative Models Linear Models for Classification Oliver Schulte - CMPT 726 Bishop PRML Ch. 4 Discriminant Functions Generative Models Discriminative Models Classification: Hand-written Digit

1.45k views • 95 slides
Models w/ Latent Random Variables Graham Neubig Site https://phontron.com/class/nn4nlp2017/

Models w/ Latent Random Variables Graham Neubig Site https://phontron.com/class/nn4nlp2017/

CS11-747 Neural Networks for NLP Models w/ Latent Random Variables Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Discriminative vs. Generative Models Discriminative model: calculate the probability of output given input

606 views • 37 slides
Models w/ Latent Random Variables Chunting Zhou Site https://phontron.com/class/nn4nlp2019/

Models w/ Latent Random Variables Chunting Zhou Site https://phontron.com/class/nn4nlp2019/

CS11-747 Neural Networks for NLP Models w/ Latent Random Variables Chunting Zhou Site https://phontron.com/class/nn4nlp2019/ With Slides from Graham Neubig Discriminative vs. Generative Models Discriminative model: calculate the

921 views • 62 slides
HiddenVariable Models for Discriminative Reranking Terry Koo and Michael Collins {

HiddenVariable Models for Discriminative Reranking Terry Koo and Michael Collins {

HiddenVariable Models for Discriminative Reranking Terry Koo and Michael Collins { maestro|mcollins } @csail.mit.edu Overview of reranking The reranking approach Use a baseline model to get the N -best candidates Rerank the candidates

1.15k views • 53 slides
Analyzing Artifacts in Discriminative and Generative Models Richard Zhang ( ) Research

Analyzing Artifacts in Discriminative and Generative Models Richard Zhang ( ) Research

Analyzing Artifacts in Discriminative and Generative Models Richard Zhang ( ) Research Scientist, Adobe SF GAMES Webinar Aug 2020 Example classifications P(correct class) P(correct class) not Shift-Invariant Deep Networks are not

1.48k views • 118 slides
Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models Language Models Language models are generative models of text s ~ P(x) The Malfoys! said Hermione. Harry was watching him. He looked like

1.18k views • 37 slides

More recommend