enabling language models to fill in the blanks
play

Enabling Language Models to Fill in the Blanks Chris Donahue Percy - PowerPoint PPT Presentation

Nov 24, 2022 •325 likes •673 views

Enabling Language Models to Fill in the Blanks Chris Donahue Percy Liang Mina Lee Paper https://arxiv.org/abs/2005.05339 Code https://github.com/chrisdonahue/ilm Demo https://chrisdonahue.com/ilm Why filling in the blanks? Hi Chris,

  1. Enabling Language Models to Fill in the Blanks Chris Donahue Percy Liang Mina Lee Paper https://arxiv.org/abs/2005.05339 Code https://github.com/chrisdonahue/ilm Demo https://chrisdonahue.com/ilm

  2. Why filling in the blanks? Hi Chris, Thanks for updating the draft. Can you revert the wording of the task definition? Editing and revising

  3. Why filling in the blanks? Hi Chris, Thanks for updating the draft. The modifications look Can you revert the wording of the task definition? Editing and revising

  4. Why filling in the blanks? Hi Chris, Thanks for updating the draft. The modifications look great to me. Can you revert the wording of the task definition? Editing and revising

  5. Why filling in the blanks? Hi Chris, Thanks for updating the draft. The modifications look good with one exception. Can you revert the wording of the task definition? Editing and revising

  6. Why filling in the blanks? We were lost in the dark forest. Suddenly, Connecting ideas

  7. Why filling in the blanks? We were lost in the dark forest. Suddenly, a bear emerged from the trees! Connecting ideas

  8. Why filling in the blanks? We were lost in the dark forest. Suddenly, A wave of relief washed over us and we ran over to greet the other traveler. Connecting ideas

  9. Why filling in the blanks? We were lost in the dark forest. Suddenly, we saw a flashlight in the distance. A wave of relief washed over us and we ran over to greet the other traveler. Connecting ideas

  10. Text infilling She ate leftover pasta for lunch . She ate [blank] for [blank]. Output Input Given�incomplete�text�with� [blank] s,�predict�complete�text Arbitrary�number�of�blanks Variable�length�spans�(e.g.�word,�sentence,�paragraph)

  11. Previous work on text infilling She ate leftover pasta for lunch . She ate [blank] for [blank]. Output Input General-purpose models GPT-3� (Brown�et�al.,�2020):�Cannot�consider� future�context

  12. Previous work on text infilling She ate leftover pasta for lunch . She ate [mask] [mask] for [mask]. Output Input General-purpose models GPT-3� (Brown�et�al.,�2020):�Cannot�consider� future�context BERT �(Devlin�et�al.,�2019):�Must�know� exact�number�of�tokens

  13. Previous work on text infilling She ate leftover pasta for lunch . She ate [blank] for [blank]. Output Input General-purpose models GPT-3� (Brown�et�al.,�2020):�Cannot�consider� future�context BERT �(Devlin�et�al.,�2019):�Must�know� exact�number�of�tokens Task-specific models SA �(Zhu�et�al.,�2019):�Cannot�leverage�pre-trained�language�models

  14. Our Idea: Infilling by Language Modeling (ILM) 1. Download your favorite language model (LM) Language Model

  15. Our Idea: Infilling by Language Modeling (ILM) 1. Download your favorite language model (LM) 2. Fine-tune the model on infilling examples Language She ate [blank] for [blank]. [sep] leftover pasta [answer] lunch [answer] Model

  16. Our Idea: Infilling by Language Modeling (ILM) Training�time 1. Manufacture infilling examples

  17. Our Idea: Infilling by Language Modeling (ILM) Training�time 1. Manufacture infilling examples leftover pasta [answer] lunch [answer] She ate leftover pasta for lunch. She ate [blank] for [blank]. Data

  18. Our Idea: Infilling by Language Modeling (ILM) Training�time 1. Manufacture infilling examples leftover pasta [answer] lunch [answer] She ate leftover pasta for lunch. Data She ate [blank] for [blank]. Input

  19. Our Idea: Infilling by Language Modeling (ILM) Training�time 1. Manufacture infilling examples She ate leftover pasta for lunch. Data She ate [blank] for [blank]. leftover pasta [answer] lunch [answer] Input Target

  20. Our Idea: Infilling by Language Modeling (ILM) Training�time 1. Manufacture infilling examples She ate leftover pasta for lunch. Data She ate [blank] for [blank]. [sep] leftover pasta [answer] lunch [answer] New data

  21. Our Idea: Infilling by Language Modeling (ILM) Training�time 2. Download pre-trained left-to-right LM She ate [blank] for [blank]. [sep] leftover pasta [answer] lunch [answer] Language Model

  22. Our Idea: Infilling by Language Modeling (ILM) Training�time 3. Fine-tune LM on infilling examples She ate [blank] for [blank]. [sep] leftover pasta [answer] lunch [answer] Language Model

  23. Our Idea: Infilling by Language Modeling (ILM) Training�time 3. Fine-tune LM on infilling examples Language She ate [blank] for [blank]. [sep] leftover pasta [answer] lunch [answer] Model

  24. Our Idea: Infilling by Language Modeling (ILM) Test�time Use fine-tuned LM to infill He drinks [blank] after [blank]. [sep] Input Language Model

  25. Our Idea: Infilling by Language Modeling (ILM) Test�time Use fine-tuned LM to infill He drinks [blank] after [blank]. [sep] water [answer] running [answer] Input Target Language Model

  26. Our Idea: Infilling by Language Modeling (ILM) Test�time Use fine-tuned LM to infill He drinks [blank] after [blank]. [sep] water [answer] running [answer] Input Target He drinks water after running. Output

  27. Experimental setup Data Stories�(Mostafazadeh�et�al.,�2016),�Abstracts,�Lyrics Metric Score,�Perplexity Model BERT,�SA�(Zhu�et�al.,�2019),�LM,�ILM�(ours) 1. Human evaluation 2. Quantitative evaluation

  28. 1. Human evaluation: Turing test Identify one of the five sentences generated by machine. Patty was excited about having her friends over. She had been working hard preparing the food. She also had the place looking spotless. All of her friends arrived and were seated at the table. Patty had a great time with her friends.

  29. 1. Human evaluation: Turing test Identify one of the five sentences generated by machine. Patty was excited about having her friends over. She had been working hard preparing the food. She also had the place looking spotless. All of her friends arrived and were seated at the table. Patty had a great time with her friends.

  30. 1. Human evaluation: Turing test Identify one of the five sentences generated by machine. Patty was excited about having her friends over. She had been working hard preparing the food. She also had the place looking spotless. [blank] All of her friends arrived and were seated at the table. Patty had a great time with her friends.

  31. 1. Human evaluation: Turing test Identify one of the five sentences generated by machine. Patty was excited about having her friends over. She had been working hard preparing the food. She also had the place looking spotless. [blank] All of her friends arrived and were seated at the table. Patty had a great time with her friends. ILM Patty knew her friends wanted pizza.

  32. 1. Human evaluation: Turing test Identify one of the five sentences generated by machine. favoritea ", Mary brightly said. BERT 20% Patty was excited about having her friends over. She wasn't sure she had to go to the store. SA 29% She had been working hard preparing the food. She also had the place looking spotless. [blank] She went to check the tv. LM 41% All of her friends arrived and were seated at the table. Patty had a great time with her friends. ILM 45% Patty knew her friends wanted pizza.

  33. 2. Quantitative evaluation Stories Abstracts Lyrics LM 18.3 27.9 27.7 ILM 15.6 22.4 22.6 Perplexity on the sentence infilling task Take�advantage�of� bidirectional�context �despite�using� unidirectional�models� Please�refer�to�the�paper�for�more�experiments�and�detailed�analysis

  34. Takeaways Conceptual simplicity Minimal�change�to�standard�LM�training Model-agnostic framework Leverage�massively�pre-trained�LMs Thank you for watching ! Thank [blank] for [blank]! Output Input Paper https://arxiv.org/abs/2005.05339 Code https://github.com/chrisdonahue/ilm Demo https://chrisdonahue.com/ilm

Recommend

CLASS IX CHAPTER 7 INTRODUCTION TO PRESENTATION SOFTWARE 1. Fill in the blanks:- a. Slide b.

CLASS IX CHAPTER 7 INTRODUCTION TO PRESENTATION SOFTWARE 1. Fill in the blanks:- a. Slide b.

CLASS IX CHAPTER 7 INTRODUCTION TO PRESENTATION SOFTWARE 1. Fill in the blanks:- a. Slide b. Title c. Template d. Insert e. View f. Ctrl + S g. .pptx h. Normal i. Design j. Design 2. Write True or False: - a. False b. True c. True d. False e.

447 views • 5 slides
Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution ? What is language? never start with a dictionary definition!! the language of Google search query completion A language is a dialect with

1.13k views • 30 slides
4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language models: log-linear language models [9, 4], which take a very di ff erent approach than the count-based n -grams described above. 11 4.1 Model

502 views • 8 slides
D b) in, of, from, at, on c) of, from, in, on, at d) of, from, in, at, on e) in, of, from, on,

D b) in, of, from, at, on c) of, from, in, on, at d) of, from, in, at, on e) in, of, from, on,

(PUC-PR) Choose the correct alternative to fill in the blanks: I What are you thinking ________? II He died ________ the injuries caused by a terrible accident. III They succeeded ________ breaking the door open. IV Everybody laughed

540 views • 15 slides
Semantic Conceptual Models Hans-Georg Fill Co-sponsored by the Austrian Science Fund: Grant Number:

Semantic Conceptual Models Hans-Georg Fill Co-sponsored by the Austrian Science Fund: Grant Number:

Workshop on Graphical Modeling Language Development July 3, 2012 Kgs. Lyngby, Denmark SeMFIS: A Tool for Managing Semantic Conceptual Models Hans-Georg Fill Co-sponsored by the Austrian Science Fund: Grant Number: J3028-N23 Agenda

686 views • 20 slides
CRA Cyber-Security Collaborative Research Alliance: MACRO: Models for Enabling Continuous

CRA Cyber-Security Collaborative Research Alliance: MACRO: Models for Enabling Continuous

CRA Cyber-Security Collaborative Research Alliance: MACRO: Models for Enabling Continuous Reconfigurability of Secure Missions Prof. Patrick McDaniel Initial Feedback Meeting ARL - Adelphi, MD October 15th, 2013 CRA: Models for Enabling

435 views • 7 slides
Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models answer the question: How likely is a string of English words good English? Help with reordering p lm (the house is small) > p lm (small the is

1.06k views • 51 slides
Mathematics for Computing Equational Logic as Calculus of Boolean Expressions COMP SCI 1FC3

Mathematics for Computing Equational Logic as Calculus of Boolean Expressions COMP SCI 1FC3

Axioms Fill in the Blanks! Anything Wrong? false (3.1) Axiom, Associativity of : = (3.15) p p false (3.2) Axiom, Symmetry of : p p (3.3) Axiom, Identity of : = (3.11) p q p q

487 views • 4 slides
Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models Language Models Language models are generative models of text s ~ P(x) The Malfoys! said Hermione. Harry was watching him. He looked like

1.18k views • 37 slides
Tsutomu Shoki and Takahiro Onoue HOYA Corporation, Blanks Division EUV blank roadmap

Tsutomu Shoki and Takahiro Onoue HOYA Corporation, Blanks Division EUV blank roadmap

Fiducial mark development Tsutomu Shoki and Takahiro Onoue HOYA Corporation, Blanks Division EUV blank roadmap Fidcuial mark process development Introduction Status and plan IEUVI Mask TWG: Feb. 12, 2012 1 EUV blanks roadmap

410 views • 5 slides
CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan

CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan

CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan Klein, Michael Collins, Yejin Choi, Dan Jurafsky Overview The language modeling problem N-gram language models Evaluation: perplexity

530 views • 50 slides
Spot the Differences Find the 4 differences between the images on the next slide Answers Fill in

Spot the Differences Find the 4 differences between the images on the next slide Answers Fill in

Spot the Differences Find the 4 differences between the images on the next slide Answers Fill in the Blanks _____ Cell _____ Cell 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. Answers Animal Cell Plant Cell 1. Chloroplast 2. Cell wall

87 views • 8 slides
Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for

Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for

Statistical language models Statistical language models in IR Discussion Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for IR Hinrich Sch utze Institute for Natural Language Processing,

1.69k views • 141 slides
Genesis and present status of restoration practice in saline blanks in mangrove wetlands Regional

Genesis and present status of restoration practice in saline blanks in mangrove wetlands Regional

Genesis and present status of restoration practice in saline blanks in mangrove wetlands Regional Colloquium on Mangrove Restoration Chennai 30-31 Aug. 2012 Dr.V.Selvam M.S.Swaminathan Research Foundation Chennai Saline blanks common in arid

531 views • 24 slides
3 Language Models 1: n -gram Language Models While the final goal of a statistical machine

3 Language Models 1: n -gram Language Models While the final goal of a statistical machine

3 Language Models 1: n -gram Language Models While the final goal of a statistical machine translation system is to create a model of the target sentence E given the source sentence F , P ( E | F ), in this chapter we will take a step back, and

106 views • 8 slides
3 Language Models 1: n -gram Language Models While the final goal of a probabilistic machine

3 Language Models 1: n -gram Language Models While the final goal of a probabilistic machine

3 Language Models 1: n -gram Language Models While the final goal of a probabilistic machine translation system is to create a model of the target sentence E given the source sentence F , P ( E | F ), in this chapter we will take a step back, and

458 views • 8 slides
Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some

Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some

Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some slides adapted from 3SLP Recap from last time Maximum Entropy Models a more general language model argmax ) ()

1.46k views • 66 slides
CSE 490 Natural Language Processing Spring 2016 Language Models Yejin Choi Slides adapted from

CSE 490 Natural Language Processing Spring 2016 Language Models Yejin Choi Slides adapted from

CSE 490 Natural Language Processing Spring 2016 Language Models Yejin Choi Slides adapted from Dan Klein, Michael Collins, Luke Zettlemoyer, Dan Jurafsky Overview The language modeling problem N-gram language models Evaluation:

991 views • 68 slides
CSE 447/547 Natural Language Processing Winter 2020 Language Models Yejin Choi Slides adapted

CSE 447/547 Natural Language Processing Winter 2020 Language Models Yejin Choi Slides adapted

CSE 447/547 Natural Language Processing Winter 2020 Language Models Yejin Choi Slides adapted from Dan Klein, Michael Collins, Luke Zettlemoyer, Dan Jurafsky Overview The language modeling problem N-gram language models Evaluation:

1.14k views • 65 slides
Building the next Generation of MapReduce Programming Models over MPI to Fill the Gaps between

Building the next Generation of MapReduce Programming Models over MPI to Fill the Gaps between

Building the next Generation of MapReduce Programming Models over MPI to Fill the Gaps between Data Analytics and Supercomputers Michela Taufer University of Delaware Collaborators: Tao Gao, Boyu Zhang (University of Delaware) Pavan Balaji,

856 views • 33 slides
Language Models Dan Klein, John DeNero UC Berkeley Language Models Language Models Acoustic

Language Models Dan Klein, John DeNero UC Berkeley Language Models Language Models Acoustic

Language Models Dan Klein, John DeNero UC Berkeley Language Models Language Models Acoustic Confusions the station signs are in deep in english -14732 the stations signs are in deep in english -14735 the station signs are in deep into

1.44k views • 86 slides
Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic

Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic

Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic Confusions the station signs are in deep in english -14732 the stations signs are in deep in english -14735 the station signs are in deep into

757 views • 22 slides
Introduction So far: Point-wise classification (geometric models) Whats next: Structured

Introduction So far: Point-wise classification (geometric models) Whats next: Structured

University of Oslo : Department of Informatics INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Probabilities and Language Models Stephan Oepen & Milen Kouylekov Language Technology Group (LTG) October 15,

556 views • 27 slides
Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and

Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and

Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and Computation, U. of Amsterdam MSc Brain & Cognitive Science, Artificial Intelligence, Logic MoL Guest Lecture Computational Models of Language

945 views • 58 slides

More recommend