Language Models Machine Translation Lecture 3 Instructor: Chris - PowerPoint PPT Presentation

Language Models Machine Translation Lecture 3 Instructor: Chris Callison-Burch TAs: Mitchell Stern, Justin Chiu Website: mt-class.org/penn

No MT yet • Today we will talk about models of p (sentence) • The rest of this semester will deal with   p (translated sentence | input sentence) • Why do it this way? • Conditioning on more stuff makes modeling more complicated. That is: p (sentence) is easier than p (translated sentence | input sentence). • Language models are arguably the most important models in statistical MT

My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her. If you want to know why I am always spleening her, it is because I am always elsewhere with friends, and disseminating so much currency, and performing so many things that can spleen a mother. Father used to dub me Shapka, for the fur hat I would don even in the summer month. He ceased dubbing me that because I ordered him to cease dubbing me that. It sounded boyish to me, and I have always thought of myself as very potent and generative. I have many many girls, believe me, and they all have a different name for me. One dubs me Baby, not because I am a baby, but because she attends to me.

Language Models Matter • Language models play the role of ... • a judge of grammaticality • a judge of semantic plausibility • an enforcer of stylistic consistency • a repository of knowledge (?)

What is the probability of a sentence? • Requirements • Assign a probability to every sentence (i.e., string of words) • Questions • How many sentences are there in English? • Too many :)

What is the probability of a sentence? • Requirements • Assign a probability to every sentence (i.e., string of words) • Questions X p LM ( e ) = 1 • How many sentences are there in e ∈ Σ ∗ English? p LM ( e ) ≥ 0 ∀ e ∈ Σ ∗ • Too many :)

Why do we want to estimate the probability of a sentence? • Goal: Assign a higher probability to good sentences in English p LM (the house is small) > p LM (small the is house) translations of German Haus: home, house … p LM (I am going home) > p LM (I am going house)

n -gram LMs p LM ( e ) = p ( e 1 , e 2 , e 3 , . . . , e ` ) p ( e 1 ) × = p ( e 2 | e 1 ) × p ( e 3 | e 1 , e 2 ) × Vector-valued random variable p ( e 4 | e 1 , e 2 , e 3 ) × · · · × p ( e ` | e 1 , e 2 , . . . , e ` − 2 , e ` − 1 )

n -gram LMs p LM ( e ) = p ( e 1 , e 2 , e 3 , . . . , e ` ) p ( e 1 ) × ≈ p ( e 2 | e 1 ) × p ( e 3 | e 1 , e 2 ) × p ( e 4 | e 1 , e 2 , e 3 ) × · · · × p ( e ` | e 1 , e 2 , . . . , e ` − 2 , e ` − 1 )

Is the Markov assumption valid for Language? • the old man are/is • the pictures are/is • The old man in the pictures is my dad.

n -gram LMs p LM ( e ) = p ( e 1 , e 2 , e 3 , . . . , e ` ) p LM ( e ) = p ( e 1 , e 2 , e 3 , . . . , e ` ) p ( e 1 ) × p ( e 1 ) × ≈ ≈ p ( e 2 | e 1 ) × p ( e 2 | e 1 ) × p ( e 3 | e 1 , e 2 ) × p ( e 3 | e 1 , e 2 ) × p ( e 4 | e 1 , e 2 , e 3 ) × p ( e 4 | e 1 , e 2 , e 3 ) × · · · × · · · × p ( e ` | e 1 , e 2 , . . . , e ` − 2 , e ` − 1 ) p ( e ` | e 1 , e 2 , . . . , e ` − 2 , e ` − 1 ) Which do you think is better? Why?

Language Models Machine Translation Lecture 3 Instructor: Chris - PowerPoint PPT Presentation

Language Models Machine Translation Lecture 3 Instructor: Chris Callison-Burch TAs: Mitchell Stern, Justin Chiu Website: mt-class.org/penn No MT yet Today we will talk about models of p (sentence) The rest of this semester will deal

Models of Language Evolution models thereof its evolution language Models of Language Evolution

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

Chapter 7 Language models Statistical Machine Translation Language models Language models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan

Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for

3 Language Models 1: n -gram Language Models While the final goal of a statistical machine

3 Language Models 1: n -gram Language Models While the final goal of a probabilistic machine

Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some

CSE 447/547 Natural Language Processing Winter 2020 Language Models Yejin Choi Slides adapted

CSE 490 Natural Language Processing Spring 2016 Language Models Yejin Choi Slides adapted from

Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic

Language Models Dan Klein, John DeNero UC Berkeley Language Models Language Models Acoustic

Introduction So far: Point-wise classification (geometric models) Whats next: Structured

Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and

III.4 Statistical Language Models 1. Basics of Statistical Language Models 2. Query-Likelihood

Language models Chapter 3 in Martin/Jurafsky Probabilistic Language Models Goal: assign a

N-grams & Language ID If N-gram models represent language models, can we use N-gram

Language Models: Evaluation & Neural Models CMSC 470 Marine Carpuat Slides credit: Jurasky

Language Models Philipp Koehn 8 September 2020 Philipp Koehn Machine Translation: Language

Models and The Unified Modeling Language (UML) What is a Model: Maps Models are abstractions of

Language Models (2) CMSC 470 Marine Carpuat Slides credit: Jurasky & Martin Roadmap

CS54701: Information Retrieval CS-54701 Information Retrieval Retrieval Models: Language models

Chapter 4: Advanced IR Models 4.1 Probabilistic IR 4.2 Statistical Language Models (LMs) 4.3

Language Models Machine Translation Lecture 3 Instructor: Chris - PowerPoint PPT Presentation

Language Models Machine Translation Lecture 3 Instructor: Chris Callison-Burch TAs: Mitchell Stern, Justin Chiu Website: mt-class.org/penn No MT yet Today we will talk about models of p (sentence) The rest of this semester will deal

Models of Language Evolution models thereof its evolution language Models of Language Evolution

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

Chapter 7 Language models Statistical Machine Translation Language models Language models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan

Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for

3 Language Models 1: n -gram Language Models While the final goal of a statistical machine

3 Language Models 1: n -gram Language Models While the final goal of a probabilistic machine

Maxent Models (III), &amp; Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some

CSE 447/547 Natural Language Processing Winter 2020 Language Models Yejin Choi Slides adapted

CSE 490 Natural Language Processing Spring 2016 Language Models Yejin Choi Slides adapted from

Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic

Language Models Dan Klein, John DeNero UC Berkeley Language Models Language Models Acoustic

Introduction So far: Point-wise classification (geometric models) Whats next: Structured

Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and

III.4 Statistical Language Models 1. Basics of Statistical Language Models 2. Query-Likelihood

Language models Chapter 3 in Martin/Jurafsky Probabilistic Language Models Goal: assign a

N-grams &amp; Language ID If N-gram models represent language models, can we use N-gram

Language Models: Evaluation &amp; Neural Models CMSC 470 Marine Carpuat Slides credit: Jurasky

Language Models Philipp Koehn 8 September 2020 Philipp Koehn Machine Translation: Language

Models and The Unified Modeling Language (UML) What is a Model: Maps Models are abstractions of

Language Models (2) CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin Roadmap

CS54701: Information Retrieval CS-54701 Information Retrieval Retrieval Models: Language models

Chapter 4: Advanced IR Models 4.1 Probabilistic IR 4.2 Statistical Language Models (LMs) 4.3

Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some

N-grams & Language ID If N-gram models represent language models, can we use N-gram

Language Models: Evaluation & Neural Models CMSC 470 Marine Carpuat Slides credit: Jurasky

Language Models (2) CMSC 470 Marine Carpuat Slides credit: Jurasky & Martin Roadmap