measuring immediate adaptation performance for neural
play

Measuring Immediate Adaptation Performance for Neural Machine - PowerPoint PPT Presentation

Mar 20, 2024 •233 likes •765 views

Measuring Immediate Adaptation Performance for Neural Machine Translation Patrick Simianer , Joern Wuebker, John DeNero Lilt NAACL 19 Outline Motivation & Approach 1 2 Evaluation Conclusion 3 2 / 20 Motivation Online adaptation is

  1. Measuring Immediate Adaptation Performance for Neural Machine Translation Patrick Simianer , Joern Wuebker, John DeNero Lilt NAACL ’19

  2. Outline Motivation & Approach 1 2 Evaluation Conclusion 3 2 / 20

  3. Motivation Online adaptation is a key feature of modern computer-aided translation (CAT) 3 / 20

  4. Motivation Online adaptation is a key feature of modern computer-aided translation (CAT) Non-adaptive system Der Terrier beißt die Frau Source #1: 3 / 20

  5. Motivation Online adaptation is a key feature of modern computer-aided translation (CAT) Non-adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: 3 / 20

  6. Motivation Online adaptation is a key feature of modern computer-aided translation (CAT) Non-adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier bites the woman Reference #1: 3 / 20

  7. Motivation Online adaptation is a key feature of modern computer-aided translation (CAT) Non-adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier bites the woman Reference #1: Der Mann beißt den Terrier Source #2: 3 / 20

  8. Motivation Online adaptation is a key feature of modern computer-aided translation (CAT) Non-adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier bites the woman Reference #1: Der Mann beißt den Terrier Source #2: The dog bites the man Hypothesis #2: 3 / 20

  9. Motivation Online adaptation is a key feature of modern computer-aided translation (CAT) Non-adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier bites the woman Reference #1: Der Mann beißt den Terrier Source #2: The dog bites the man Hypothesis #2: The man bites the terrier Reference #2: 3 / 20

  10. Motivation Translators have a reasonable expectation that . . . 1 New vocabulary (in context) gets quickly picked up by the system, ideally right away 2 The system generally adapts to new domains 4 / 20

  11. Motivation Translators have a reasonable expectation that . . . 1 New vocabulary (in context) gets quickly picked up by the system, ideally right away 2 The system generally adapts to new domains With neural machine translation fine-tuning can readily be used [Turchi et al., 2017] ( inter-alia ): θ i ← θ i − 1 − γ ∇L ( θ i − 1 , x i , y i ) . 4 / 20

  12. Approach • Typically [Turchi et al., 2017, Peris et al., 2017, Bertoldi et al., 2014] ( inter-alia ) fine-tuning is evaluated in a batch setting • Corpus BLEU or isolated sentence-wise metrics are often used • These do not necessarily express how fast a system adapts 5 / 20

  13. Approach • Typically [Turchi et al., 2017, Peris et al., 2017, Bertoldi et al., 2014] ( inter-alia ) fine-tuning is evaluated in a batch setting • Corpus BLEU or isolated sentence-wise metrics are often used • These do not necessarily express how fast a system adapts As we will show this is not good enough → We seek to measure perceived, immediate adaptation performance 5 / 20

  14. Approach Calculate recall on the set of all words that are not stopwords, ignoring length [Papineni et al., 2002] and ordering issues 1 [Kothur et al., 2018] 1 In each of the data sets considered in this work, the average number of occurrences of content words ranges between 1.01 and 1.11 per sentence 6 / 20

  15. Approach Calculate recall on the set of all words that are not stopwords, ignoring length [Papineni et al., 2002] and ordering issues 1 [Kothur et al., 2018] Since the task is online adaptation — specifically focus on few-shot learning : Consider only first and second occurrences of words! 1 In each of the data sets considered in this work, the average number of occurrences of content words ranges between 1.01 and 1.11 per sentence 6 / 20

  16. One-Shot Recall R1 After seeing a word exactly once before in a reference/confirmed translation, is it correctly produced the second time around? 7 / 20

  17. One-Shot Recall R1 After seeing a word exactly once before in a reference/confirmed translation, is it correctly produced the second time around? R1 i = |H i ∩ R 1 , i | |R 1 , i | H i : Content words in the hypothesis i th example Content words whose second occurrence is in R 1 , i : the reference for i th example 7 / 20

  18. One-Shot Recall R1: Example Adaptive system Der Terrier beißt die Frau Source #1: 8 / 20

  19. One-Shot Recall R1: Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: 8 / 20

  20. One-Shot Recall R1: Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier bites the woman Reference #1: 8 / 20

  21. One-Shot Recall R1: Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier bites the woman Reference #1: R1=0/0 8 / 20

  22. One-Shot Recall R1: Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier bites the woman Reference #1: R1=0/0 Der Mann beißt den Terrier Source #2: 8 / 20

  23. One-Shot Recall R1: Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier bites the woman Reference #1: R1=0/0 Der Mann beißt den Terrier Source #2: The terrier bites the man Hypothesis #2: 8 / 20

  24. One-Shot Recall R1: Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier bites the woman Reference #1: R1=0/0 Der Mann beißt den Terrier Source #2: The terrier bites the man Hypothesis #2: The man bites 1 the terrier 1 Reference #2: 8 / 20

  25. One-Shot Recall R1: Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier bites the woman Reference #1: R1=0/0 Der Mann beißt den Terrier Source #2: The terrier bites the man Hypothesis #2: The man bites 1 the terrier 1 Reference #2: R1=2 / 2 8 / 20

  26. One-Shot Recall R1: Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier bites the woman Reference #1: R1=0/0 Der Mann beißt den Terrier Source #2: The terrier bites the man Hypothesis #2: The man bites 1 the terrier 1 Reference #2: R1=2 / 2 R1=2 / 2 Total: 8 / 20

  27. Zero-Shot Recall R0 Not having seen a word before, is it still correctly produced? Is the system adapting to the domain at hand? 9 / 20

  28. Zero-Shot Recall R0 Not having seen a word before, is it still correctly produced? Is the system adapting to the domain at hand? R0 i = |H i ∩ R 0 , i | |R 0 , i | H i : Content words in the hypothesis for i th example Content words that occur for the first time in the R 0 , i : reference for i th example 9 / 20

  29. Zero- and One-Shot Recall R0+1 Combined metric. R0+1 i = |H i ∩ [ R 0 , i ∪ R 1 , i ] | |R 0 , i ∪ R 1 , i | H i : Content words in the hypothesis for i th example Content words that occur for the first or second R 0 , i ∪ R 1 , i : time in the reference for i th example 10 / 20

  30. Corpus-Level Metric � |G| i = 1 |H i ∩ R 0 , i | R 0 Corpus = � |G| i = 1 |R 0 , i | Corpus of |G| source, reference/confirmed seg- G : ment, hypothesis triplets 11 / 20

  31. Complete Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier 0 bites 0 the woman 0 Reference #1: R1=0 / 0 12 / 20

  32. Complete Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier 0 bites 0 the woman 0 Reference #1: R1=0 / 0 R0=1 / 3 12 / 20

  33. Complete Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier 0 bites 0 the woman 0 Reference #1: R1=0 / 0 R0=1 / 3 R0+1=1 / 3 12 / 20

  34. Complete Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier 0 bites 0 the woman 0 Reference #1: R1=0 / 0 R0=1 / 3 R0+1=1 / 3 Der Mann beißt den Terrier Source #2: 12 / 20

  35. Complete Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier 0 bites 0 the woman 0 Reference #1: R1=0 / 0 R0=1 / 3 R0+1=1 / 3 Der Mann beißt den Terrier Source #2: The terrier bites the man Hypothesis #2: 12 / 20

  36. Complete Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier 0 bites 0 the woman 0 Reference #1: R1=0 / 0 R0=1 / 3 R0+1=1 / 3 Der Mann beißt den Terrier Source #2: The terrier bites the man Hypothesis #2: The man 0 bites 1 the terrier 1 Reference #2: 12 / 20

  37. Complete Example Adaptive system Der Terrier beißt die Frau Source #1: The dog bites the lady Hypothesis #1: The terrier 0 bites 0 the woman 0 Reference #1: R1=0 / 0 R0=1 / 3 R0+1=1 / 3 Der Mann beißt den Terrier Source #2: The terrier bites the man Hypothesis #2: The man 0 bites 1 the terrier 1 Reference #2: R1=2 / 2 12 / 20

Recommend

NEPR208 - Adaptation properties and mechanisms Functional advantages in properties of a neural

NEPR208 - Adaptation properties and mechanisms Functional advantages in properties of a neural

NEPR208 - Adaptation properties and mechanisms Functional advantages in properties of a neural code and changes in those properties What is adaptation? Why do neural systems have a particular nonlinear and filter? Why does the nonlinearity and

931 views • 38 slides
Community Based Measuring and Prioritizing Adaptation Actions in Agriculture Sector of the

Community Based Measuring and Prioritizing Adaptation Actions in Agriculture Sector of the

Community Based Measuring and Prioritizing Adaptation Actions in Agriculture Sector of the Gangetic Basin SVRK Prabhakar With study partners in Bangladesh, India and Nepal Institute for Global Environmental Strategies Japan Presented to the

1.18k views • 65 slides
Key Performance I ndicators in Measuring PAI s & PLO Performance For Electrical Engineering

Key Performance I ndicators in Measuring PAI s & PLO Performance For Electrical Engineering

Key Performance I ndicators in Measuring PAI s & PLO Performance For Electrical Engineering Department Curriculum of Malaysian Polytechnics (The he role o of KPIs Is to assess a and improve the he Ou Outcomes-based E Educati tion)

741 views • 47 slides
Measuring the Adaptation of Countries to Societal Aging John W. Rowe, MD Julius Richmond

Measuring the Adaptation of Countries to Societal Aging John W. Rowe, MD Julius Richmond

Measuring the Adaptation of Countries to Societal Aging John W. Rowe, MD Julius Richmond Professor of Health Policy & Ageing Columbia University Mailman School of Public Health Organization for Economic Cooperation and Development Paris,

560 views • 31 slides
Measuring and Evaluating Computer System Performance CSE 141, S2'06 Jeff Brown Performance

Measuring and Evaluating Computer System Performance CSE 141, S2'06 Jeff Brown Performance

Measuring and Evaluating Computer System Performance CSE 141, S2'06 Jeff Brown Performance Marches On ... But what is performance ? CSE 141, S2'06 Jeff Brown The bottom line: Performance Time to Throughput Car Speed Passengers Bay

274 views • 23 slides
Measuring Performance: Chapter 4! Or My computer is faster than your computer with thanks to

Measuring Performance: Chapter 4! Or My computer is faster than your computer with thanks to

Measuring Performance: Chapter 4! Or My computer is faster than your computer with thanks to Larry Carter, UCSD 1 Performance Marches On ... 1200 DEC Alpha 21264/600 1100 1000 900 800 700 Performance 600 500 DEC Alpha 5/500 400

728 views • 29 slides
Measuring Adaptation Between Dialogs

Measuring Adaptation Between Dialogs

Measuring Adaptation Between Dialogs

591 views • 38 slides
How to Get Good Performance by Using OpenMP ! ! Loop optimizations ! ! Measuring OpenMP

How to Get Good Performance by Using OpenMP ! ! Loop optimizations ! ! Measuring OpenMP

Agenda ! How to Get Good Performance by Using OpenMP ! ! Loop optimizations ! ! Measuring OpenMP performance ! ! Best practices ! ! Task parallelism !! !" #" Memory access patterns ! Correctness versus performance ! It

166 views • 14 slides
Performance measures Nicolas Papageorgiou AQF-2005 Measuring hedge fund performance What

Performance measures Nicolas Papageorgiou AQF-2005 Measuring hedge fund performance What

Performance measures Nicolas Papageorgiou AQF-2005 Measuring hedge fund performance What is the best method? Risk adjusted performance measures Multi-factor models Peer group factors Return based factor Asset based

416 views • 24 slides
Measuring Performance November 17, 2008 Measuring Performance Introduction CPU Peformance and

Measuring Performance November 17, 2008 Measuring Performance Introduction CPU Peformance and

Introduction CPU Peformance and Its Factors Evaluating Performance Measuring Performance November 17, 2008 Measuring Performance Introduction CPU Peformance and Its Factors Evaluating Performance Outline 1 Introduction 2 CPU Peformance and

988 views • 65 slides
Measuring Social Performance in LAC Micol Pistelli, Director of Social Performance, MIX December

Measuring Social Performance in LAC Micol Pistelli, Director of Social Performance, MIX December

Microfinance Information Exchange Measuring Social Performance in LAC Micol Pistelli, Director of Social Performance, MIX December 2012 The Premier Source for Microfinance Data and Analysis This presentation is the proprietary and/or

790 views • 6 slides
A Neural Model of Adaptation in Reading Marten van Schijndel and Tal Linzen November 4, 2018

A Neural Model of Adaptation in Reading Marten van Schijndel and Tal Linzen November 4, 2018

A Neural Model of Adaptation in Reading Marten van Schijndel and Tal Linzen November 4, 2018 Department of Cognitive Science, Johns Hopkins University van Schijndel and Linzen November 4, 2018 1 / 24 Thought experiment cassowary

1.06k views • 53 slides
Measuring Political Bias in British Media: Using Recurrent Neural Networks for Long Form Textual

Measuring Political Bias in British Media: Using Recurrent Neural Networks for Long Form Textual

Measuring Political Bias in British Media: Using Recurrent Neural Networks for Long Form Textual Analysis Rory How May 29, 2020 1 Introduction Measuring Political Bias in British Media: 2/42 Rory How May 29, 2020 Using Recurrent Neural

527 views • 42 slides
Measuring Well Being and Performance: Measuring Well-Being and Performance: Purpose, Measures and

Measuring Well Being and Performance: Measuring Well-Being and Performance: Purpose, Measures and

UCL ENERGY INSTITUTE UCL ENERGY INSTITUTE UCL ENERGY INSTITUTE UCL ENERGY INSTITUTE UCL ENERGY INSTITUTE Measuring Well Being and Performance: Measuring Well-Being and Performance: Purpose, Measures and Policy A presentation to the

603 views • 19 slides
Measuring Entanglement Negativity with Neural Network Estimators Johnnie Gray , Leonardo

Measuring Entanglement Negativity with Neural Network Estimators Johnnie Gray , Leonardo

Measuring Entanglement Negativity with Neural Network Estimators Johnnie Gray , Leonardo Banchi , , Abolfazl Bayat , Sougato Bose 6 Nov 2017 Imperial College, London, UK University College London, London, UK Machine

464 views • 28 slides
Neural Network Meets DCN: Traffic-driven Topology Adaptation with Deep Learning Moewi Wang, Yong

Neural Network Meets DCN: Traffic-driven Topology Adaptation with Deep Learning Moewi Wang, Yong

Neural Network Meets DCN: Traffic-driven Topology Adaptation with Deep Learning Moewi Wang, Yong Cui, Shihan Xiao, Xin wang, Dan Yang, Kai Chen, Jun Zhu Introduction Conventional wired data centers generally adopt a static network topology

338 views • 14 slides
Measuring the Impact of Docker on Network I/O Performance

Measuring the Impact of Docker on Network I/O Performance

University of Amsterdam Masters Programme in System and Network Engineering Measuring the Impact of Docker on Network I/O Performance Author: Rohprimardho

363 views • 12 slides
Coastal Adaptation Kellie Fisher FCERM Senior Advisor Why Adaptation? Adaptation to a

Coastal Adaptation Kellie Fisher FCERM Senior Advisor Why Adaptation? Adaptation to a

Coastal Adaptation Kellie Fisher FCERM Senior Advisor Why Adaptation? Adaptation to a changing environment. Coastal change is not new! Traditional defences are a form of adaptation too. Are they always appropriate? Property

516 views • 19 slides
Measuring and Reasoning About Performance Readings: 1.4-1.5 1 Goals for this Class

Measuring and Reasoning About Performance Readings: 1.4-1.5 1 Goals for this Class

Measuring and Reasoning About Performance Readings: 1.4-1.5 1 Goals for this Class Understand how CPUs run programs How do we express the computation the CPU? How does the CPU execute it? How does the CPU support other

552 views • 26 slides
Measuring and Reasoning About Performance Readings: 1.4-1.5 1 Goals for this Class

Measuring and Reasoning About Performance Readings: 1.4-1.5 1 Goals for this Class

Measuring and Reasoning About Performance Readings: 1.4-1.5 1 Goals for this Class Understand how CPUs run programs How do we express the computation the CPU? How does the CPU execute it? How does the CPU support other

1.38k views • 122 slides
Measuring and Reasoning About Performance Readings: 1.4-1.5 1 Goals for this Class

Measuring and Reasoning About Performance Readings: 1.4-1.5 1 Goals for this Class

Measuring and Reasoning About Performance Readings: 1.4-1.5 1 Goals for this Class Understand how CPUs run programs How do we express the computation the CPU? How does the CPU execute it? How does the CPU support other

1.35k views • 114 slides
In-Test Adaptation of Workload in Enterprise Application Performance Testing Maciej Kaczmarski

In-Test Adaptation of Workload in Enterprise Application Performance Testing Maciej Kaczmarski

In-Test Adaptation of Workload in Enterprise Application Performance Testing Maciej Kaczmarski April 23, 2017 Agenda 1 Motivation & Research Objective 2 Proposed Approach 3 Experimental Evaluation 4 Conclusions & Future work Maciej

481 views • 12 slides
Measuring Parallel Performance How well does my application scale? Funding Partners bioexcel.eu

Measuring Parallel Performance How well does my application scale? Funding Partners bioexcel.eu

Measuring Parallel Performance How well does my application scale? Funding Partners bioexcel.eu 1 Reusing this material This work is licensed under a Creative Commons Attribution- NonCommercial-ShareAlike 4.0 International License.

819 views • 25 slides
Java Performance : Measuring and Tuning Nataraja Neelakanta Overview Most of the applications

Java Performance : Measuring and Tuning Nataraja Neelakanta Overview Most of the applications

Java Performance : Measuring and Tuning Nataraja Neelakanta Overview Most of the applications today are designed and developed for simultaneous use by a large number of users. At times there are underlying problems in the code,

509 views • 27 slides

More recommend