bertscore evaluating text generation with bert
play

BERTScore: Evaluating Text Generation with BERT Varsha Kishore - PowerPoint PPT Presentation

Mar 26, 2023 •780 likes •1.02k views

BERTScore: Evaluating Text Generation with BERT Varsha Kishore Tianyi Zhang Felix Wu Kilian Q. Weinberger Yoav Artzi I am like I like translate ich liebe es I like it I love it I am loving it I am like Candidate Reference I like

  1. BERTScore: Evaluating Text Generation with BERT Varsha Kishore Tianyi Zhang Felix Wu Kilian Q. Weinberger Yoav Artzi

  2. I am like I like translate ich liebe es I like it I love it I am loving it

  3. I am like Candidate Reference I like translate ich liebe es I like it I love it I love it I am loving it 0.88/1.00 Metric

  4. Text Generation Evaluation Metrics N-gram matching Embedding-based approaches metrics BLEU (Papineni et al., 2002) Meant 2.0 (Lo, 2017) METEOR (Banerjee & Lavie, 2005) YiSi -1 (Lo et al., 2018) ROUGE (Lin, 2004) BERTScore chrF (Popovic, 2015)

  5. BLEU N-gram Matching Reference 
 The weather is cold today Candidate 1 
 Candidate 2 
 The weather is sunny today It is freezing today BLEU cannot identify synonyms BLEU gives higher score to candidate 1

  6. BERTScore: an evaluation metric that uses BERT embeddings

  7. BERT Transformer model pre-trained on 
 masked language modeling and next sentence prediction Generates word token embeddings that reflect their context

  8. BERTScore the weather is cold today Reference the weather is cold today Pairwise cosine similarity Candidate it is freezing today Contextual it is freezing today embedding

  9. Greedy Matching the weather Candidate is ≈ ç cold today is freezing today it ≈ ç Reference

  10. Greedy Matching Precision Recall Match words in candidate to reference Match words in reference to candidate

  11. Greedy Matching Precision Recall Match words in candidate to reference Match words in reference to candidate

  12. Greedy Matching Precision Recall Match words in candidate to reference Match words in reference to candidate

  13. Greedy Matching Precision Recall Match words in candidate to reference Match words in reference to candidate

  14. Greedy Matching Precision Recall 0.713 0.713 0.515 0.858 0.858 0.796 0.796 0.913 0.913 Match words in candidate to reference Match words in reference to candidate

  15. Greedy Matching - Aggregate Precision Recall 0.713 0.515 0.858 0.796 0.713 0.858 0.796 0.913 0.913

  16. Greedy Matching - Aggregate Precision Recall 0.713 0.515 0.858 0.796 0.713 0.858 0.796 0.913 0.913 0.759 0.820

  17. <latexit sha1_base64="ft0u0d5gLkLIktmgMFQEAFxGQ3o=">ACRnicbZDNaxNBGMbfTW0bo61rPXoZDIghN1SUJBCQCgeo5gPyC5hdvJuMmT2g5l3S8Oy4P/mpWdv/glePLSIV2ezOWjiCwMPz/O8zMwvypU05HnfndbBg8Oj4/bDzqPHJ6dP3KdnI5MVWuBQZCrTk4gbVDLFIUlSOMk18iRSOI5W7+t8fI3ayCz9TOscw4QvUhlLwclaMzcMCG+ovPIrdsnOWfAuiDUXZeMONApZr1YsEPOMWGN/QsGVqr91udxsztej1vM2xf+FvRhe0MZu63YJ6JIsGUhOLGTH0vp7DkmqRQWHWCwmDOxYovcGplyhM0YbnBULGX1pmzONP2pMQ27t8bJU+MWSeRbSaclmY3q83/ZdOC4rdhKdO8IExFc1FcKEYZq5myubQISK2t4EJL+1YmltxyJEu+YyH4u1/eF6Pznu/1/I8X3f7oS4OjDc/hBbwCH95AHz7AIYg4Cv8gDu4d26dn84v53dTbTlbhM/gn2nBHwO+s9M=</latexit> <latexit sha1_base64="ft0u0d5gLkLIktmgMFQEAFxGQ3o=">ACRnicbZDNaxNBGMbfTW0bo61rPXoZDIghN1SUJBCQCgeo5gPyC5hdvJuMmT2g5l3S8Oy4P/mpWdv/glePLSIV2ezOWjiCwMPz/O8zMwvypU05HnfndbBg8Oj4/bDzqPHJ6dP3KdnI5MVWuBQZCrTk4gbVDLFIUlSOMk18iRSOI5W7+t8fI3ayCz9TOscw4QvUhlLwclaMzcMCG+ovPIrdsnOWfAuiDUXZeMONApZr1YsEPOMWGN/QsGVqr91udxsztej1vM2xf+FvRhe0MZu63YJ6JIsGUhOLGTH0vp7DkmqRQWHWCwmDOxYovcGplyhM0YbnBULGX1pmzONP2pMQ27t8bJU+MWSeRbSaclmY3q83/ZdOC4rdhKdO8IExFc1FcKEYZq5myubQISK2t4EJL+1YmltxyJEu+YyH4u1/eF6Pznu/1/I8X3f7oS4OjDc/hBbwCH95AHz7AIYg4Cv8gDu4d26dn84v53dTbTlbhM/gn2nBHwO+s9M=</latexit> <latexit sha1_base64="ft0u0d5gLkLIktmgMFQEAFxGQ3o=">ACRnicbZDNaxNBGMbfTW0bo61rPXoZDIghN1SUJBCQCgeo5gPyC5hdvJuMmT2g5l3S8Oy4P/mpWdv/glePLSIV2ezOWjiCwMPz/O8zMwvypU05HnfndbBg8Oj4/bDzqPHJ6dP3KdnI5MVWuBQZCrTk4gbVDLFIUlSOMk18iRSOI5W7+t8fI3ayCz9TOscw4QvUhlLwclaMzcMCG+ovPIrdsnOWfAuiDUXZeMONApZr1YsEPOMWGN/QsGVqr91udxsztej1vM2xf+FvRhe0MZu63YJ6JIsGUhOLGTH0vp7DkmqRQWHWCwmDOxYovcGplyhM0YbnBULGX1pmzONP2pMQ27t8bJU+MWSeRbSaclmY3q83/ZdOC4rdhKdO8IExFc1FcKEYZq5myubQISK2t4EJL+1YmltxyJEu+YyH4u1/eF6Pznu/1/I8X3f7oS4OjDc/hBbwCH95AHz7AIYg4Cv8gDu4d26dn84v53dTbTlbhM/gn2nBHwO+s9M=</latexit> <latexit sha1_base64="ft0u0d5gLkLIktmgMFQEAFxGQ3o=">ACRnicbZDNaxNBGMbfTW0bo61rPXoZDIghN1SUJBCQCgeo5gPyC5hdvJuMmT2g5l3S8Oy4P/mpWdv/glePLSIV2ezOWjiCwMPz/O8zMwvypU05HnfndbBg8Oj4/bDzqPHJ6dP3KdnI5MVWuBQZCrTk4gbVDLFIUlSOMk18iRSOI5W7+t8fI3ayCz9TOscw4QvUhlLwclaMzcMCG+ovPIrdsnOWfAuiDUXZeMONApZr1YsEPOMWGN/QsGVqr91udxsztej1vM2xf+FvRhe0MZu63YJ6JIsGUhOLGTH0vp7DkmqRQWHWCwmDOxYovcGplyhM0YbnBULGX1pmzONP2pMQ27t8bJU+MWSeRbSaclmY3q83/ZdOC4rdhKdO8IExFc1FcKEYZq5myubQISK2t4EJL+1YmltxyJEu+YyH4u1/eF6Pznu/1/I8X3f7oS4OjDc/hBbwCH95AHz7AIYg4Cv8gDu4d26dn84v53dTbTlbhM/gn2nBHwO+s9M=</latexit> F1 = 2 Precision · Recall Precision + Recall

  18. Reference the weather is cold today F1 Score Candidate it is freezing today Pairwise Contextual cosine embedding similarity

  19. Evaluation: WMT Translation Benchmark Human Metric Reference: The weather is cold today. 0.85 0.77 Candidate: It is freezing today. compute correlation Reference: The garden is nice. 0.77 0.71 Candidate: The garden was pretty. Reference: I like apples very much. 0.80 0.79 Candidate: I love apples.

  20. Correlation Study 0.8 BLEU ITER YiSi-1 RUSE BertScore F1 0.6 Correlation 0.4 0.2 0 Czech-English German-English English-Czech English-German Language Pair

  21. 4 tasks 8 languages 363 systems

  22. Download here :https://pypi.org/project/bert-score/ Or Just: pip install bert_score Github

Recommend

What can Statistical Machine Translation teach Neural Text Generation about Optimization? Graham

What can Statistical Machine Translation teach Neural Text Generation about Optimization? Graham

What can Statistical Machine Translation teach Neural Text Generation about Optimization? Graham Neubig @ NAACL Workshop on Methods for Optimizing and Evaluating Neural Language Generation 6/6/2019 or How to Optimize your Neural Generation

638 views • 34 slides
GANocracy Outline Background: Text Generation Latent-Variable Generation Learning

GANocracy Outline Background: Text Generation Latent-Variable Generation Learning

Controlling Text Generation Alexander Rush (and Sam Wiseman) Harvard / Cornell Tech GANocracy Outline Background: Text Generation Latent-Variable Generation Learning Neural Templates Machine Learning for Text Generation y 1: T =

710 views • 55 slides
Text-to-Text Generation Katja Filippova katjaf@google.com Friday, August 19, 2011 1 This

Text-to-Text Generation Katja Filippova katjaf@google.com Friday, August 19, 2011 1 This

Text-to-Text Generation Katja Filippova katjaf@google.com Friday, August 19, 2011 1 This course A quick overview of a number of topics under the umbrella term text-to-text generation. Research problems - what is being done and

1.95k views • 119 slides
BERT 3.0 The New BERT Wheres Ernie????? Logging into Bert BERT now uses the same style logon as

BERT 3.0 The New BERT Wheres Ernie????? Logging into Bert BERT now uses the same style logon as

BERT 3.0 The New BERT Wheres Ernie????? Logging into Bert BERT now uses the same style logon as the other SEIMS+ applications. BERT Main Screen Icons on Main Screen of BERT There are several icons on the main screen for BERT. These icons are

590 views • 16 slides
Unsupervised Machine Translation Sachin Kumar Conditional Text Generation Generate text

Unsupervised Machine Translation Sachin Kumar Conditional Text Generation Generate text

CMU CS11-737: Multilingual NLP (Fall 2020) Unsupervised Machine Translation Sachin Kumar Conditional Text Generation Generate text according to a specification: P(Y|X) Input X Output Y (Text) Task English Hindi Machine Translation

743 views • 35 slides
11-830 Computational Ethics for NLP Ethical Concerns on OpenAI Text Generation System Discussion

11-830 Computational Ethics for NLP Ethical Concerns on OpenAI Text Generation System Discussion

11-830 Computational Ethics for NLP Ethical Concerns on OpenAI Text Generation System Discussion OpenAI Text Generation System CNN: This AI is so good at writing that its creators wont let you use it Takes in a text prompt and

336 views • 8 slides
BERT Bidirectional Encoder Representations from Transformers Introduction What is BERT?

BERT Bidirectional Encoder Representations from Transformers Introduction What is BERT?

BERT Bidirectional Encoder Representations from Transformers Introduction What is BERT? Latest language representational model BERT is conceptually simple and empirically powerful. One of the biggest challenges in natural language

839 views • 20 slides
Text-to-Image Generation Yu Cheng Text-to-Image Synthesis Text-to-Image Synthesis

Text-to-Image Generation Yu Cheng Text-to-Image Synthesis Text-to-Image Synthesis

Text-to-Image Generation Yu Cheng Text-to-Image Synthesis Text-to-Image Synthesis StackGAN, AttnGAN, TAGAN, ObjGAN Text-to-Video Synthesis GAN-based methods, VAE-based methods, StoryGAN Dialogue-based Image Synthesis

827 views • 45 slides
From Text Message to Research Paper: Teaching the Text Message Generation Heidi Wright Ohio

From Text Message to Research Paper: Teaching the Text Message Generation Heidi Wright Ohio

From Text Message to Research Paper: Teaching the Text Message Generation Heidi Wright Ohio State University Introduction and Organization Origins- SIUC 2006 Increased number of tasks and shorter time frame Insufficient background

241 views • 22 slides
Parsing Eric McCreath Overview In this lecture we will look at: structured text, generation,

Parsing Eric McCreath Overview In this lecture we will look at: structured text, generation,

Parsing Eric McCreath Overview In this lecture we will look at: structured text, generation, parsing, tokens, grammars, and writing a simple parser. 2 Structured Text Information is often stored in text files. Also we often provide

522 views • 13 slides
Non-Monotonic Sequential Text Generation Sean Welleck, Kiant Brantley, Hal Daum III,

Non-Monotonic Sequential Text Generation Sean Welleck, Kiant Brantley, Hal Daum III,

Non-Monotonic Sequential Text Generation Sean Welleck, Kiant Brantley, Hal Daum III, Kyunghyun Cho Sequential Text Generation Y = ( y 1 , y 2 , , y N ) ( hi , how , are , you , ? ) Sequential Text Generation Unconditional Y ( hi ,

459 views • 24 slides
Statistical Perspectives on Text-to-Text Generation Noah Smith Language Technologies Institute

Statistical Perspectives on Text-to-Text Generation Noah Smith Language Technologies Institute

Statistical Perspectives on Text-to-Text Generation Noah Smith Language Technologies Institute Machine Learning Department School of Computer Science Carnegie Mellon University nasmith@cs.cmu.edu Im A Learning Guy I use statistics for

1.2k views • 82 slides
Text Generation with Exemplar-based Adaptive Decoding Hao Peng, Ankur Parikh, Manaal Faruqui,

Text Generation with Exemplar-based Adaptive Decoding Hao Peng, Ankur Parikh, Manaal Faruqui,

Text Generation with Exemplar-based Adaptive Decoding Hao Peng, Ankur Parikh, Manaal Faruqui, Bhuwan Dhingra, Dipanjan Das @NAACL June 4, 2019 Outline Background and Overview Adaptive Decoding Experiments Conditioned Text Generation

652 views • 49 slides
Speech Generation From Concept and from Text Martin Jansche CS 6998 2004-02-11 Components of

Speech Generation From Concept and from Text Martin Jansche CS 6998 2004-02-11 Components of

Speech Generation From Concept and from Text Martin Jansche CS 6998 2004-02-11 Components of spoken output systems Front end: From input to control parameters. From naturally occurring text; or From constrained mark-up language; or

420 views • 27 slides
Intractable Peacebuilding: Evaluating a Generation of Work Across the Israeli-Palestinian Divide

Intractable Peacebuilding: Evaluating a Generation of Work Across the Israeli-Palestinian Divide

Intractable Peacebuilding: Evaluating a Generation of Work Across the Israeli-Palestinian Divide Ned Lazarus Network for Peacebuilding Evaluation Thursday Talk May 21, 2015 Presentation Research Overview: Four Evaluative Studies

471 views • 17 slides
Automatically Evaluating Text Coherence Using Discourse Relations Ziheng Lin , Hwee Tou Ng and

Automatically Evaluating Text Coherence Using Discourse Relations Ziheng Lin , Hwee Tou Ng and

Automatically Evaluating Text Coherence Using Discourse Relations Ziheng Lin , Hwee Tou Ng and Min-Yen Kan Department of Computer Science National University of Singapore Introduction Textual coherence discourse structure Conditional

702 views • 36 slides
Evaluating Haptic and Auditory Guidance to Assist Blind People in Reading Printed Text Using

Evaluating Haptic and Auditory Guidance to Assist Blind People in Reading Printed Text Using

Evaluating Haptic and Auditory Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras TACCES ESS | ASSET ETS 2016 Lee Stearns 1 , Ruofei Du 1 , Uran Oh 1 , Catherine Jou 1 , Leah Findlater 2 , David A. Ross 3 , Jon

870 views • 83 slides
Learning to generate: Concept-to-text generation using machine learning Ioannis Konstas

Learning to generate: Concept-to-text generation using machine learning Ioannis Konstas

Learning to generate: Concept-to-text generation using machine learning Ioannis Konstas Institute for Language, Cognition and Computation University of Edinburgh Aberdeen, NLG Summer School 21 July 2015 Konstas (ILCC) Concept-to-Text

1.83k views • 143 slides
MaskGAN: Better Text Generation via Filling in the ______ June 5, 2018 (

MaskGAN: Better Text Generation via Filling in the ______ June 5, 2018 (

MaskGAN: Better Text Generation via Filling in the ______ June 5, 2018 ( ) Sungjae Cho (Interdisciplinary Program in Cognitive Science) sj.cho@snu.ac.kr SNU Spoken Language Processing Lab /

502 views • 49 slides
Controllable Response Generation Susana Benavidez Andrew Kirjner Nick Seay Mentor: Sina

Controllable Response Generation Susana Benavidez Andrew Kirjner Nick Seay Mentor: Sina

Controllable Response Generation Susana Benavidez Andrew Kirjner Nick Seay Mentor: Sina Semnani Overview Part 1 Text Generation vs Controllable Text Generation Part 2 Conditional Training Weighted Decoding Part 3 Transformer + Attribute

741 views • 39 slides
Unsupervised Concept-to-text Generation with Hypergraphs Ioannis Konstas, Mirella Lapata

Unsupervised Concept-to-text Generation with Hypergraphs Ioannis Konstas, Mirella Lapata

Unsupervised Concept-to-text Generation with Hypergraphs Ioannis Konstas, Mirella Lapata Institute for Language, Cognition and Computation University of Edinburgh NAACL 2012, Montral Konstas, Lapata (ILCC) Unsupervised Concept-to-text

1.47k views • 62 slides
Architecture in Motion How Adyen achieved 100x Bert Wolters - EVP Technology bert@adyen.com

Architecture in Motion How Adyen achieved 100x Bert Wolters - EVP Technology bert@adyen.com

Architecture in Motion How Adyen achieved 100x Bert Wolters - EVP Technology bert@adyen.com Traditional vs Today Customers are in full control $1B 80% On Singles Day in China, $1Billion was On Black Friday in the U.S., nearly 80% of

1.44k views • 46 slides
T h e P o w e r o f S u p e r g r a v i t y S o l u t i o n s Bert

T h e P o w e r o f S u p e r g r a v i t y S o l u t i o n s Bert

T h e P o w e r o f S u p e r g r a v i t y S o l u t i o n s Bert Vercnocke University of Amsterdam GGI, 8 September 2016 O v e r v i e w 1. Intro 2. Singularity resolution: Black hole microstates

700 views • 31 slides
GANs for Word Embeddings Akshay Budhkar and Krishnapriya Introduction GANs have shown incredible

GANs for Word Embeddings Akshay Budhkar and Krishnapriya Introduction GANs have shown incredible

GANs for Word Embeddings Akshay Budhkar and Krishnapriya Introduction GANs have shown incredible quality w/ generation of images Discrete nature of text makes it harder to train generation of text GANs for Text Some ways people

128 views • 8 slides

More recommend