more is less non parametric language models and efficiency
play

More is Less? Non-parametric Language Models and Efficiency Graham - PowerPoint PPT Presentation

Sep 10, 2022 •141 likes •833 views

Carnegie Mellon University More is Less? Non-parametric Language Models and Efficiency Graham Neubig Carnegie Mellon University Based on Work by Junxian He w/ Taylor Berg-Kirkpatrick 1 Carnegie Mellon University Parametric Language Models

  1. Carnegie Mellon University More is Less? Non-parametric Language Models and Efficiency Graham Neubig Carnegie Mellon University Based on Work by Junxian He w/ Taylor Berg-Kirkpatrick 1

  2. Carnegie Mellon University Parametric Language Models Parametric LM <s> 2

  3. Carnegie Mellon University Parametric Language Models I Parametric LM <s> 3

  4. Carnegie Mellon University Parametric Language Models I Parametric LM <s> I 4

  5. Carnegie Mellon University Parametric Language Models I ordered Parametric LM <s> I 5

  6. Carnegie Mellon University Parametric Language Models I ordered a Parametric LM <s> I ordered 6

  7. Carnegie Mellon University Parametric Language Models I ordered a pizza with sauce . </s> Parametric LM <s> I ordered a pizza with sauce . 7

  8. Carnegie Mellon University Non-Parametric Language Models Non-Parametric LM Non-parametric datastore (typically training dataset) 8

  9. Carnegie Mellon University Non-Parametric Language Models Non-Parametric LM Prototype sentence I ordered a burger with fries Non-parametric datastore (typically training dataset) 9

  10. Carnegie Mellon University Non-Parametric Language Models Non-Parametric LM Prototype sentence <s> I ordered a burger with fries Non-parametric datastore (typically training dataset) 10

  11. Carnegie Mellon University Non-Parametric Language Models I Non-Parametric LM Prototype sentence <s> I ordered a burger with fries Non-parametric datastore (typically training dataset) 11

  12. Carnegie Mellon University Non-Parametric Language Models I ordered a pizza with sauce . </s> Non-Parametric LM Prototype sentence <s> I ordered a pizza with sauce . I ordered a burger with fries Non-parametric datastore (typically training dataset) 12

  13. Carnegie Mellon University Non-Parametric Language Models I ordered a pizza with sauce . </s> Non-Parametric LM Prototype sentence <s> I ordered a pizza with sauce . I ordered a burger with fries Non-parametric datastore (typically training dataset) 13

  14. Carnegie Mellon University A Human Processing Analogy 14

  15. Carnegie Mellon University A Human Processing Analogy Our Language Ability/ Knowledge 14

  16. Carnegie Mellon University A Human Processing Analogy Our Language Ability/ Our Language Ability/ Knowledge + Knowledge Wikipedia/Google 14

  17. Carnegie Mellon University A (Very Incomplete) Survey of Neural Non-parametric Models for NLP 15

  18. Carnegie Mellon University A (Very Incomplete) Survey of Neural Non-parametric Models for NLP •Count-based non-parametric data stores: • Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016. • Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016. • Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017. 15

  19. Carnegie Mellon University A (Very Incomplete) Survey of Neural Non-parametric Models for NLP •Count-based non-parametric data stores: • Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016. • Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016. • Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017. •Bias probabili6es based on count-based aggrega6on of retrieved sentences: •Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018. •Deep Weighted Averaging Classifiers. Card et al. 2019. •Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020. 15

  20. Carnegie Mellon University A (Very Incomplete) Survey of Neural Non-parametric Models for NLP •Count-based non-parametric data stores: • Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016. • Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016. • Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017. •Bias probabili6es based on count-based aggrega6on of retrieved sentences: •Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018. •Deep Weighted Averaging Classifiers. Card et al. 2019. •Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020. •A;end to retrieved sentences: • Search Engine Guided Non-Parametric Neural Machine Transla/on. Gu et al. 2018. • Genera/ng Sentences by Edi/ng Prototypes. Guu et al. 2018. • REALM: Retrieval-Augmented Language Model Pre-Training. Guu et al. 2020. 15

  21. Carnegie Mellon University A (Very Incomplete) Survey of Neural Non-parametric Models for NLP •Count-based non-parametric data stores: • Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016. • Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016. • Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017. •Bias probabili6es based on count-based aggrega6on of retrieved sentences: •Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018. •Deep Weighted Averaging Classifiers. Card et al. 2019. •Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020. •A;end to retrieved sentences: • Search Engine Guided Non-Parametric Neural Machine Transla/on. Gu et al. 2018. • Genera/ng Sentences by Edi/ng Prototypes. Guu et al. 2018. • REALM: Retrieval-Augmented Language Model Pre-Training. Guu et al. 2020. •Feed retrieved states into model: •Learning to Remember Rare Events. Kaiser et al. 2017. 15

  22. Carnegie Mellon University A (Very Incomplete) Survey of Neural Non-parametric Models for NLP •Count-based non-parametric data stores: • Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016. • Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016. • Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017. •Bias probabili6es based on count-based aggrega6on of retrieved sentences: •Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018. •Deep Weighted Averaging Classifiers. Card et al. 2019. •Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020. •A;end to retrieved sentences: • Search Engine Guided Non-Parametric Neural Machine Transla/on. Gu et al. 2018. • Genera/ng Sentences by Edi/ng Prototypes. Guu et al. 2018. • REALM: Retrieval-Augmented Language Model Pre-Training. Guu et al. 2020. •Feed retrieved states into model: •Learning to Remember Rare Events. Kaiser et al. 2017. •Fine-tune models on retrieved sentences: •One Sentence One Model for Neural Machine Transla/on. Li et al. 2016. 15

  23. Carnegie Mellon University Non-Parametric Language Models: Pros/Cons Pros • Micgate pressure on the parametric models • Improved interpretability in the modeling process

  24. Carnegie Mellon University Non-Parametric Language Models: Pros/Cons Pros • Micgate pressure on the parametric models • Improved interpretability in the modeling process Cons • In almost all cases, large parametric datastore leads to significant issues with memory and speed efficiency at test cme

  25. Carnegie Mellon University A Concrete Example: Prototype-based Language Models (Guu et al. TACL 2018) 18

Recommend

The Significance of Errors to Parametric Models of Language Acquisition Paula Buttery Natural

The Significance of Errors to Parametric Models of Language Acquisition Paula Buttery Natural

The Significance of Errors to Parametric Models of Language Acquisition Paula Buttery Natural Language and Information Processing Group Computer Laboratory, Cambridge University paula.buttery@cl.cam.ac.uk Paula Buttery, 03/2004 Classification

395 views • 16 slides
Semi-parametric and response setup non-parametric approaches to Parametric models

Semi-parametric and response setup non-parametric approaches to Parametric models

Christian Ritz Concentration- Semi-parametric and response setup non-parametric approaches to Parametric models concentration-response modelling Non- parametric approach Semi- parametric Christian Ritz models Results University of

492 views • 23 slides
Parametric vs Nonparametric Models Parametric models assume some finite set of parameters .

Parametric vs Nonparametric Models Parametric models assume some finite set of parameters .

Parametric vs Nonparametric Models Parametric models assume some finite set of parameters . Given the parameters, future predictions, x , are independent of the observed data, D : P ( x | , D ) = P ( x | ) therefore capture

679 views • 5 slides
Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution ? What is language? never start with a dictionary definition!! the language of Google search query completion A language is a dialect with

1.13k views • 30 slides
Models for Probability Distributions and Density Functions 1 General Concepts Parametric:

Models for Probability Distributions and Density Functions 1 General Concepts Parametric:

Models for Probability Distributions and Density Functions 1 General Concepts Parametric: E.g., Gaussian, Gamma, Binomial Non-Parametric: E.g., kernel estimates Intermediate models: Mixture Models 2 Gaussian Mixture Model

382 views • 22 slides
4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language models: log-linear language models [9, 4], which take a very di ff erent approach than the count-based n -grams described above. 11 4.1 Model

501 views • 8 slides
h ( n ) w ( n ) x ( n ) Linear nonparametric vs. parametric models Many researchers use

h ( n ) w ( n ) x ( n ) Linear nonparametric vs. parametric models Many researchers use

Linear Signal Models Overview Introduction Introduction h ( n ) w ( n ) x ( n ) Linear nonparametric vs. parametric models Many researchers use signal models to analyze stationary Equivalent representations univariate time series

507 views • 5 slides
Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models answer the question: How likely is a string of English words good English? Help with reordering p lm (the house is small) &gt; p lm (small the is

1.06k views • 51 slides
Testing for Parametric Orderings Efficiency Sergio Ortobelli 1 , 3 Nikolas Topaloglou 4 Matteo

Testing for Parametric Orderings Efficiency Sergio Ortobelli 1 , 3 Nikolas Topaloglou 4 Matteo

Introduction &amp; Aims and Motivations Classification of Choices Testing Methodology Conclusions References Testing for Parametric Orderings Efficiency Sergio Ortobelli 1 , 3 Nikolas Topaloglou 4 Matteo Malavasi 2 , 1 2 Department of Actuarial

786 views • 23 slides
Chapter 3 Parametric Models and Methods Models: Weibull ( includes the exponential model)

Chapter 3 Parametric Models and Methods Models: Weibull ( includes the exponential model)

Chapter 3 Parametric Models and Methods Models: Weibull ( includes the exponential model) log-normal log-logistic gamma Remark: Except for the gamma distribution, these lifetime dis- tributions have the property that the

731 views • 34 slides
Stochastic Model Efficiency Applications: Cluster-Distance Sampling and Parametric Curve Fitting

Stochastic Model Efficiency Applications: Cluster-Distance Sampling and Parametric Curve Fitting

Stochastic Model Efficiency Applications: Cluster-Distance Sampling and Parametric Curve Fitting to Tackle Sampling Errors and Bias Yvonne C. Chueh, PhD, ASA Paul H. Johnson, Jr., PhD Yongxue Qi Joint work between the University of Illinois at

525 views • 37 slides
Discovery of Inflectional Paradigms from Plain Text using Graphical Models over Strings Markus

Discovery of Inflectional Paradigms from Plain Text using Graphical Models over Strings Markus

A Non-Parametric Model for the Discovery of Inflectional Paradigms from Plain Text using Graphical Models over Strings Markus Dreyer Center for Language and Speech Processing (CLSP) Human Language Technology Center of Excellence (HLTCOE) Johns

2.43k views • 200 slides
Tutorial on Parametric Timed Automata for RT Scheduling Examples of scheduling models Giuseppe

Tutorial on Parametric Timed Automata for RT Scheduling Examples of scheduling models Giuseppe

Tutorial on Parametric Timed Automata for RT Scheduling Examples of scheduling models Giuseppe Lipari and tienne Andr CRIStAL - Universit de Lille Outline Models Example 1: response time calculation Example 2: Sensitivity on

555 views • 27 slides
Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models Language Models Language models are generative models of text s ~ P(x) The Malfoys! said Hermione. Harry was watching him. He looked like

1.18k views • 37 slides
Non-Parametric Models Review of last class: Decision Tree Learning dealing with the

Non-Parametric Models Review of last class: Decision Tree Learning dealing with the

Non-Parametric Models Review of last class: Decision Tree Learning dealing with the overlearning problem: pruning ensemble learning boosting Agenda Nearest neighbor models Finding nearest neighbors with kd trees

493 views • 27 slides
Automatic Generation of Minimal and Reduced Models for Structured Parametric Dynamical Systems

Automatic Generation of Minimal and Reduced Models for Structured Parametric Dynamical Systems

Automatic Generation of Minimal and Reduced Models for Structured Parametric Dynamical Systems Igor Pontes Duff Joint work with Peter Benner (MPI, Magdeburg) Pawan Goyal (MPI, Magdeburg) ICERM Workshop - Mathematics of Reduced Order Models

1.46k views • 108 slides
Combining parametric and nonparametric models for off-policy evaluation Omer Gottesman 1 , Yao Liu

Combining parametric and nonparametric models for off-policy evaluation Omer Gottesman 1 , Yao Liu

Combining parametric and nonparametric models for off-policy evaluation Omer Gottesman 1 , Yao Liu 2 , Scott Sussex 1 , Emma Brunskill 2 , Finale Doshi-Velez 1 1 Paulson School of Engineering and Applied Science, Harvard University 2 Department of

221 views • 19 slides
CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan

CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan

CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan Klein, Michael Collins, Yejin Choi, Dan Jurafsky Overview The language modeling problem N-gram language models Evaluation: perplexity

530 views • 50 slides
Parametric Polymorphism and Abstract Models of Storage (In memory of Christopher Strachey,

Parametric Polymorphism and Abstract Models of Storage (In memory of Christopher Strachey,

Parametric Polymorphism and Abstract Models of Storage (In memory of Christopher Strachey, 1916-1975) Uday S. Reddy 1 1 University of Birmingham Strachey-100, Oxford My introduction to Stracheys ideas Section 1 Introducing the terms

530 views • 40 slides
Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for

Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for

Statistical language models Statistical language models in IR Discussion Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for IR Hinrich Sch utze Institute for Natural Language Processing,

1.69k views • 141 slides
3 Language Models 1: n -gram Language Models While the final goal of a probabilistic machine

3 Language Models 1: n -gram Language Models While the final goal of a probabilistic machine

3 Language Models 1: n -gram Language Models While the final goal of a probabilistic machine translation system is to create a model of the target sentence E given the source sentence F , P ( E | F ), in this chapter we will take a step back, and

458 views • 8 slides
3 Language Models 1: n -gram Language Models While the final goal of a statistical machine

3 Language Models 1: n -gram Language Models While the final goal of a statistical machine

3 Language Models 1: n -gram Language Models While the final goal of a statistical machine translation system is to create a model of the target sentence E given the source sentence F , P ( E | F ), in this chapter we will take a step back, and

106 views • 8 slides
Estimation risk for the VaR of portfolios driven by semi-parametric multivariate models Christian

Estimation risk for the VaR of portfolios driven by semi-parametric multivariate models Christian

General framework Estimating the conditional VaR Numerical comparison of the different VaR estimators Estimation risk for the VaR of portfolios driven by semi-parametric multivariate models Christian Francq Jean-Michel Zakoan CREST and

1.26k views • 83 slides
Estimation risk for the VaR of portfolios driven by semi-parametric multivariate models Christian

Estimation risk for the VaR of portfolios driven by semi-parametric multivariate models Christian

General framework Estimating the conditional VaR Numerical comparison of the different VaR estimators Estimation risk for the VaR of portfolios driven by semi-parametric multivariate models Christian Francq Jean-Michel Zakoan CREST and

810 views • 77 slides

More recommend