representing huge translation models statistical machine
play

Representing Huge Translation Models Statistical Machine - PowerPoint PPT Presentation

Jul 05, 2023 •162 likes •1.27k views

Representing Huge Translation Models Statistical Machine Translation parallel text + alignment Statistical Machine Translation extract rules parallel text + alignment Statistical Machine Translation score extract rules rules parallel

  1. Suffix Arrays it makes him and it mars him . it sets him on and it takes him off . # 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Text T 3 12 2 15 10 6 0 4 8 13 1 5 16 11 9 14 7 17 18 Suffix Array SA O ( | w | log | T | ) (Manber & Myers, 93) O ( | w | + log | T | ) him and it O ( | w | ) (Abouelhoda et al., 04) Query Pattern w

  2. Suffix Arrays it makes him and it mars him . it sets him on and it takes him off . # 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Text T 3 12 2 15 10 6 0 4 8 13 1 5 16 11 9 14 7 17 18 Suffix Array SA O ( | w | log | T | ) (Manber & Myers, 93) O ( | w | + log | T | ) him and it O ( | w | ) (Abouelhoda et al., 04) on baseline model: Query Pattern w 0.009 seconds/sentence (not including extraction/scoring)

  3. Problem: Phrases with Gaps • Hierarchical phrase-based translation (Chiang 2005, 2007) • Quirk et al. 2005, Simard et al. 2005, DeNeefe et al. 2007 Input it persuades him and it disheartens him Source Phrase it X him

  4. Hierarchical Phrases: Phrases with Gaps • Hierarchical phrase-based translation (Chiang 2005, 2007) • Quirk et al. 2005, Simard et al. 2005, DeNeefe et al. 2007 Input it persuades him and it disheartens him Source Phrase it X him

  5. Hierarchical Phrases: Phrases with Gaps • Hierarchical phrase-based translation (Chiang 2005, 2007) • Quirk et al. 2005, Simard et al. 2005, DeNeefe et al. 2007 Input it persuades him and it disheartens him Source Phrase it X him

  6. Hierarchical Phrases: Phrases with Gaps • Hierarchical phrase-based translation (Chiang 2005, 2007) • Quirk et al. 2005, Simard et al. 2005, DeNeefe et al. 2007 Input it persuades him and it disheartens him Source Phrase it X him

  7. Hierarchical Phrases: Phrases with Gaps • Hierarchical phrase-based translation (Chiang 2005, 2007) • Quirk et al. 2005, Simard et al. 2005, DeNeefe et al. 2007 Input it persuades him and it disheartens him Source Phrase it X and X him

  8. Problem Statement Given an input sentence, efficiently find all hierarchical phrase-based translation rules for that sentence in the training corpus.

  9. Pattern Matching for Hierachical PBMT Input Pattern it persuades him and it disheartens him

  10. Pattern Matching for Hierarchical PBMT Input Pattern it persuades him and it disheartens him it persuades him it Query Patterns persuades him and persuades him and it him and it disheartens and it disheartens him disheartens it persuades him and it persuades persuades him and it persuades him him and it disheartens him and and it disheartens him and it it persuades him and it it disheartens persuades him and it disheartens disheartens him him and it disheartens him

  11. Pattern Matching for Hierarchical PBMT Input Pattern it persuades him and it disheartens him it X and it X disheartens him Query Patterns it X it it X and X him it X disheartens persuades him X disheartens it X him persuades him X him persuades X it persuades X it disheartens persuades X disheartens persuades X disheartens him persuades X him him and X him it persuades X it him X disheartens him it persuades X disheartens it persuades him X disheartens it persuades X him it persuades him X him it X and it it persuades X it disheartens it X it disheartens it persuades X disheartens him

  12. Pattern Matching for Hierarchical PBMT Input Pattern it persuades him and it disheartens him it X and it disheartens Query Patterns it X it disheartens him persuades him and X him persuades him X disheartens him persuades X it disheartens him it persuades him and X him it persuades him X disheartens him it persuades X it disheartens him it X and it disheartens him

  13. Pattern Matching for Hierarchical PBMT Input Pattern it persuades him and it disheartens him it X and it disheartens Query Patterns it X it disheartens him persuades him and X him persuades him X disheartens him persuades X it disheartens him it persuades him and X him it persuades him X disheartens him it persuades X it disheartens him it X and it disheartens him This is a variant of approximate pattern matching (Navarro ‘01)

  14. Pattern Matching with Gaps 3 and it mars him , it sets him ... Query pattern 12 and it takes him off . # α 2 him and it mars him . it sets ... him X it him off . # 15 him on and it takes him off . # 10 him , it sets him on and it ... 6 it makes him and it mars ... 0 it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  15. Pattern Matching with Gaps 3 and it mars him , it sets him ... Query pattern α 12 and it takes him off . # 2 him and it mars him . it sets ... him X it him off . # 15 him on and it takes him off . # 10 him , it sets him on and it ... 6 it makes him and it mars ... 0 it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  16. Pattern Matching with Gaps 3 and it mars him , it sets him ... Query pattern α 12 and it takes him off . # 2 him and it mars him . it sets ... him X it him off . # 15 him on and it takes him off . # 10 him , it sets him on and it ... 6 it makes him and it mars ... 0 it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  17. Pattern Matching with Gaps 3 and it mars him , it sets him ... Query pattern 12 and it takes him off . # α 2 him and it mars him . it sets ... him X it him off . # 15 Subpatterns w i him on and it takes him off . # 10 him , it sets him on and it ... 6 him it makes him and it mars ... 0 it it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  18. Pattern Matching with Gaps 3 and it mars him , it sets him ... Query pattern 12 and it takes him off . # α 2 him and it mars him . it sets ... him X it him off . # 15 Subpatterns w i him on and it takes him off . # 10 him , it sets him on and it ... 6 him it makes him and it mars ... 0 it it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  19. Pattern Matching with Gaps 3 and it mars him , it sets him ... Query pattern 12 and it takes him off . # α 2 him and it mars him . it sets ... him X it him off . # 15 Subpatterns w i him on and it takes him off . # 10 him , it sets him on and it ... 6 him n i Occurrences it makes him and it mars ... 0 it it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  20. Pattern Matching with Gaps 3 and it mars him , it sets him ... 12 and it takes him off . # 2 him and it mars him . it sets ... him off . # 15 2 0 him on and it takes him off . # 10 15 4 him , it sets him on and it ... 6 10 8 it makes him and it mars ... 0 6 13 it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  21. Pattern Matching with Gaps 2 0 15 4 10 8 6 13

  22. Pattern Matching with Gaps (2, 4) 2 0 (2, 8) 15 4 (2, 13) 10 8 (6, 8) 6 13 (6, 13) (10, 13)

  23. Pattern Matching with Gaps (2, 4) 2 0 (2, 8) 15 4 (2, 13) 10 8 (6, 8) 6 13 (6, 13) (10, 13) RILMS (Rahman et al., 06)

  24. Pattern Matching with Gaps (2, 4) 2 0 (2, 8) 15 4 (2, 13) 10 8 (6, 8) 6 13 (6, 13) (10, 13) RILMS (Rahman et al., 06) � linear in number of occurrences of subpatterns: O ( n i ) i

  25. Baseline Timing Result 221 seconds per sentence compare: 0.009 seconds per sentence for contiguous phrases

  26. Complexity Analysis � ( | w | + log | T | ) contiguous w 137 5 27 I � � discontiguous ( | w i | + log | T | + n i ) α = w 1 X...Xw I i =1 2825 3 5 27 82069

  27. Complexity Analysis � ( | w | + log | T | ) contiguous w 137 5 27 I � � discontiguous ( | w i | + log | T | + n i ) α = w 1 X...Xw I i =1 2825 3 5 27 82069

  28. Exploiting Redundancy Input Pattern it persuades him and it disheartens him it X and it X disheartens him Query Patterns it X it it X and X him it X disheartens persuades him X disheartens it X him persuades him X him persuades X it persuades X it disheartens persuades X disheartens persuades X disheartens him persuades X him him and X him it persuades X it him X disheartens him it persuades X disheartens it persuades him X disheartens it persuades X him it persuades him X him it X and it it persuades X it disheartens it X it disheartens it persuades X disheartens him

  29. Exploiting Redundancy Input Pattern it persuades him and it disheartens him it X and it X disheartens him Query Patterns it X it it X and X him it X disheartens persuades him X disheartens it X him persuades him X him persuades X it persuades X it disheartens persuades X disheartens persuades X disheartens him persuades X him him and X him it persuades X it him X disheartens him it persuades X disheartens it persuades him X disheartens it persuades X him it persuades him X him it X and it it persuades X it disheartens it X it disheartens it persuades X disheartens him

  30. Exploiting Redundancy Query Pattern it persuades X disheartens him

  31. Exploiting Redundancy Query Pattern it persuades X disheartens him Maximal Prefix it persuades X disheartens (Zhang & Vogel 2005)

  32. Exploiting Redundancy Query Pattern it persuades X disheartens him Maximal Prefix it persuades X disheartens Maximal Suffix persuades X disheartens him

  33. Prefix Tree with Suffix Links him persuades him persuades it X him X him him

  34. Timing Results 221 seconds/ sentence Baseline

  35. Timing Results 221 177 seconds/ sentence Baseline Prefix Tree

  36. Complexity Analysis � ( | w | + log | T | ) contiguous w 137 5 27 I � � discontiguous ( | w i | + log | T | + n i ) α = w 1 X...Xw I i =1 2825 3 5 27 82069

  37. Complexity Analysis � ( | w | + log | T | ) contiguous w 137 5 27 I � � discontiguous ( | w i | + log | T | + n i ) α = w 1 X...Xw I i =1 2825 3 5 27 82069

  38. Empirical Analysis cumulative time (s) computations (ranked by time)

  39. Distribution of Patterns in Training Data Frequency Pattern types (in descending order of frequency)

  40. Distribution of Patterns in Training Data Frequency Pattern types (in descending order of frequency)

  41. Analysis of Problem • The expensive computations involve at least one frequent subpattern. There are two cases. • A frequent pattern paired with an infrequent pattern • Two frequent patterns paired with each other

  42. Frequent × Infrequent Subpatterns

  43. Frequent × Infrequent Subpatterns

  44. Frequent × Infrequent Subpatterns

  45. Frequent × Infrequent Subpatterns

  46. Double Binary Search Baeza-Yates, 04

  47. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D

  48. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D

  49. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D

  50. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D

  51. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D

  52. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D

  53. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D complexity: | Q | log | D | Upper bound

  54. Obtaining Sorted Sets

  55. Obtaining Sorted Sets Sort via Stratified Tree (van Emde Boas et al. 1977)

  56. Obtaining Sorted Sets Sort via Stratified Tree (van Emde Boas et al. 1977) Problem: complexity increases to O ( | Q | log | D | + ( | Q | + | D | ) log log | T | )

  57. Obtaining Sorted Sets Sort via Stratified Tree (van Emde Boas et al. 1977) Solution: cache sorted set in prefix tree Problem: complexity increases to O ( | Q | log | D | + ( | Q | + | D | ) log log | T | )

  58. Timing Results 221 177 seconds/ sentence Baseline Prefix + double Tree binary

  59. Timing Results 221 177 174 seconds/ sentence Baseline Prefix + double Tree binary

  60. Obtaining Sorted Sets Sort via Stratified Tree Problem: sort complexity is still very high for very frequent patterns

  61. Obtaining Sorted Sets Solution: precompute the inverted index for 1000 most frequent contiguous patterns

  62. Timing Results 221 177 174 seconds/ sentence Baseline Prefix + double Tree binary

  63. Timing Results 221 177 174 seconds/ sentence 44 Baseline Prefix + double + inverted Tree binary indices

  64. Frequent × Frequent Subpatterns

  65. Frequent × Frequent Subpatterns Problem: There is no clever algorithm to solve this problem

  66. Solution: Precomputation it makes him and it mars him . it sets him on and it takes him off . # it makes him and it mars him . it sets him on and it takes him off . # 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Text

Recommend

Chapter 11 Tree-based models Statistical Machine Translation Tree-Based Models Traditional

Chapter 11 Tree-based models Statistical Machine Translation Tree-Based Models Traditional

Chapter 11 Tree-based models Statistical Machine Translation Tree-Based Models Traditional statistical models operate on sequences of words Many translation problems can be best explained by pointing to syntax reordering, e.g., verb

1.39k views • 120 slides
Chapter 4 Word-based models Statistical Machine Translation Lexical Translation How to

Chapter 4 Word-based models Statistical Machine Translation Lexical Translation How to

Chapter 4 Word-based models Statistical Machine Translation Lexical Translation How to translate a word look up in dictionary Haus house, building, home, household, shell. Multiple translations some more frequent than others

1.2k views • 61 slides
Statistical Machine Translation George Foster George Foster Statistical Machine Translation A

Statistical Machine Translation George Foster George Foster Statistical Machine Translation A

Statistical Machine Translation George Foster George Foster Statistical Machine Translation A Brief History of MT Origins (1949): WW II codebreaking success suggests statistical approach to MT George Foster Statistical Machine Translation A

840 views • 54 slides
Statistical Machine Translation Nadir Durrani 21-November-2014 Machine Translation

Statistical Machine Translation Nadir Durrani 21-November-2014 Machine Translation

Statistical Machine Translation Nadir Durrani 21-November-2014 Machine Translation www.uni-stuttart.de Problem: Automatic translation the foreign text: 2 Open Problems in Machine Translation www.uni-stuttart.de Ambiguity in translation

939 views • 44 slides
Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models answer the question: How likely is a string of English words good English? Help with reordering p lm (the house is small) > p lm (small the is

1.06k views • 51 slides
Compositions of Tree-to-Tree Statistical Machine Translation Models Andreas Maletti Institute

Compositions of Tree-to-Tree Statistical Machine Translation Models Andreas Maletti Institute

Compositions of Tree-to-Tree Statistical Machine Translation Models Andreas Maletti Institute for Natural Language Processing University of Stuttgart, Germany Montral July 28, 2016 Compositions of Tree-to-Tree Statistical Machine

290 views • 25 slides
Statistical Machine Translation Graham Neubig Nara Institute of Science and Technology (NAIST)

Statistical Machine Translation Graham Neubig Nara Institute of Science and Technology (NAIST)

Statistical Machine Translation Statistical Machine Translation Graham Neubig Nara Institute of Science and Technology (NAIST) 10/23/2012 1 Statistical Machine Translation Machine Translation Automatically translate between languages

792 views • 42 slides
Improved Models of Distortion Cost for Statistical Machine Translation Spence Green, Michel

Improved Models of Distortion Cost for Statistical Machine Translation Spence Green, Michel

Improved Models of Distortion Cost for Statistical Machine Translation Spence Green, Michel Galley, and Christopher D. Manning Stanford University June 4, 2010 Motivation A New Cost Model Phrase-based MT Evaluation Limit v. Cost Conclusion

520 views • 39 slides
Chapter 5 Phrase-based models Statistical Machine Translation Motivation Word-Based Models

Chapter 5 Phrase-based models Statistical Machine Translation Motivation Word-Based Models

Chapter 5 Phrase-based models Statistical Machine Translation Motivation Word-Based Models translate words as atomic units Phrase-Based Models translate phrases as atomic units Advantages: many-to-many translation can handle

303 views • 27 slides
Meta-Learning for Low Resource NMT Introduction Historically Statistical Translation

Meta-Learning for Low Resource NMT Introduction Historically Statistical Translation

Meta-Learning for Low Resource NMT Introduction Historically Statistical Translation Neural Machine Translation recently outperforms Statistical Models outperformed translations on low resource language pairs NMT Previous Work

427 views • 19 slides
Statistical Machine Translation Statistical Machine Translation p Lecture 2 Theory and Praxis of

Statistical Machine Translation Statistical Machine Translation p Lecture 2 Theory and Praxis of

Components: Translation model, language model, decoder Statistical Machine Translation Lecture 2: Theory and Praxis of Decoding p Statistical Machine Translation Statistical Machine Translation p Lecture 2 Theory and Praxis of Decoding

540 views • 9 slides
Statistical Machine Translation Outline p Why Syntax? Lecture 5 Yamada and Knight:

Statistical Machine Translation Outline p Why Syntax? Lecture 5 Yamada and Knight:

Reminder: Modeling and Decoding Syntax-Based Statistical Machine Translationp Statistical Machine Translation Outline p Why Syntax? Lecture 5 Yamada and Knight: translating into trees Syntax-Based Models Wu: tree-based transfer

106 views • 7 slides
Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of Edinburgh R. Sennrich MT 2018 12 1 / 27 Todays Lecture So far, main focus of lecture was on: neural machine translation research

740 views • 46 slides
What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured Prediction? Graham Neubig @ ICLR Workshop on Deep Reinforcement Learning Meets Structured Prediction 5/6/2019 Types of Prediction Types of

1.73k views • 127 slides
Chapter 8 Evaluation Statistical Machine Translation Evaluation How good is a given machine

Chapter 8 Evaluation Statistical Machine Translation Evaluation How good is a given machine

Chapter 8 Evaluation Statistical Machine Translation Evaluation How good is a given machine translation system? Hard problem, since many different translations acceptable semantic equivalence / similarity Evaluation metrics

456 views • 42 slides
3 Language Models 1: n -gram Language Models While the final goal of a statistical machine

3 Language Models 1: n -gram Language Models While the final goal of a statistical machine

3 Language Models 1: n -gram Language Models While the final goal of a statistical machine translation system is to create a model of the target sentence E given the source sentence F , P ( E | F ), in this chapter we will take a step back, and

106 views • 8 slides
Unsupervised Morpheme Analysis Competition 3: Statistical Machine Translation Mikko Kurimo, Sami

Unsupervised Morpheme Analysis Competition 3: Statistical Machine Translation Mikko Kurimo, Sami

Unsupervised Morpheme Analysis Competition 3: Statistical Machine Translation Mikko Kurimo, Sami Virpioja, Ville T. Turunen (TKK) Graeme W. Blackwood, William Byrne (UCAM) Morphology and SMT Statistical machine translation systems find

333 views • 21 slides
Introduction to Machine Translation Joost Bastings ILLC, University of Amsterdam

Introduction to Machine Translation Joost Bastings ILLC, University of Amsterdam

Introduction to Machine Translation Joost Bastings ILLC, University of Amsterdam bastings.github.io Table of contents 1. A Brief History of MT 2. Statistical Machine Translation 3. Phrase-based Statistical Machine Translation 4. Evaluation

1.4k views • 101 slides
Statistical Machine Translation Overview p EM algorithm Lecture 3 Improved word alignment

Statistical Machine Translation Overview p EM algorithm Lecture 3 Improved word alignment

Statistical modeling Statistical Machine Translation Lecture 3: Word Alignment and Phrase Models p Statistical Machine Translation Overview p EM algorithm Lecture 3 Improved word alignment Word Alignment and Phrase Models

456 views • 8 slides
Vector Space Models for Phrase-based Machine Translation Tamer Alkhouli, Andreas Guta, and

Vector Space Models for Phrase-based Machine Translation Tamer Alkhouli, Andreas Guta, and

Vector Space Models for Phrase-based Machine Translation Tamer Alkhouli, Andreas Guta, and Hermann Ney <surname>@cs.rwth-aachen.de Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation Doha, Qatar October 25, 2014

275 views • 16 slides
Improved Word Alignments for Statistical Machine Translation Alex Fraser Institute for NLP

Improved Word Alignments for Statistical Machine Translation Alex Fraser Institute for NLP

Improved Word Alignments for Statistical Machine Translation Alex Fraser Institute for NLP University of Stuttgart Statistical Machine Translation (SMT) Build a model P( e | f ), the probability of the English sentence e given the

558 views • 28 slides
Machine Translation Classification of divergences Classical and Statistical Approaches

Machine Translation Classification of divergences Classical and Statistical Approaches

Session 4: Interlingua-based MT Dorr (1992, 1994): UNITRAN system Machine Translation Classification of divergences Classical and Statistical Approaches Lexical Conceptual Structure Translation mappings between syntactic

130 views • 12 slides
Domain Adaptation in Statistical Machine Translation Logic, Language and Computation Bart

Domain Adaptation in Statistical Machine Translation Logic, Language and Computation Bart

Domain Adaptation in Statistical Machine Translation Logic, Language and Computation Bart Mellebeek October 28, 2013 Bart Mellebeek Domain Adaptation in Statistical Machine Translation Outline Machine Translation and AI Statistical Machine

889 views • 54 slides
Using Log-linear Models for Tuning Machine Translation Output Michael Carl IAI L REC 2008 1

Using Log-linear Models for Tuning Machine Translation Output Michael Carl IAI L REC 2008 1

Using Log-linear Models for Tuning Machine Translation Output Michael Carl IAI L REC 2008 1 Overview: METIS: Architecture described in session p28 (Friday, 14:40 ) Statistical MT using: Shallow linguistic ressources (SL Analysis,

503 views • 16 slides

More recommend