the uedin systems for the iwslt 2012 evaluation
play

The UEDIN Systems for the IWSLT 2012 Evaluation Eva Hasler, Peter - PowerPoint PPT Presentation

Dec 21, 2022 •125 likes •1.16k views

The UEDIN Systems for the IWSLT 2012 Evaluation Eva Hasler, Peter Bell, Arnab Ghoshal, Barry Haddow, Philipp Koehn, Fergus McInnes, Steve Renals, Pawel Swietojanski School of Informatics University of Edinburgh December 6th Overview UEDIN

  1. Spoken Language Translation Punctuation insertion system • Phrasebased Moses, monotone decoding • Avoid excessive punctuation insertion • Only using cased instead of truecased data improved performance • Tuning sets (target: MT input) • dev2010 transcripts, dev2010+test2010 transcripts, dev2010+test2010 ASR outputs (all number-converted) • Evaluate different systems in terms of BLEU on MT source

  2. Spoken Language Translation Punctuation insertion system • Phrasebased Moses, monotone decoding • Avoid excessive punctuation insertion • Only using cased instead of truecased data improved performance • Tuning sets (target: MT input) • dev2010 transcripts, dev2010+test2010 transcripts, dev2010+test2010 ASR outputs (all number-converted) • Evaluate different systems in terms of BLEU on MT source

  3. Spoken Language Translation Punctuation insertion system • Phrasebased Moses, monotone decoding • Avoid excessive punctuation insertion • Only using cased instead of truecased data improved performance • Tuning sets (target: MT input) • dev2010 transcripts, dev2010+test2010 transcripts, dev2010+test2010 ASR outputs (all number-converted) • Evaluate different systems in terms of BLEU on MT source

  4. Spoken Language Translation SLT pipeline BLEU(MT source) test2010 ASR transcript 70.79 + number conversion 71.37 + punctuation insertion 84.80 + postprocessing 85.17 test2010 ASR out + SLT pipeline 61.82 Punctuation Insertion System BLEU(MT source) Tune: dev2010 ASR transcript test2011 ASR output + SLT pipeline 62.39 Tune: dev2010+test2010 ASR transcripts test2011 ASR output + SLT pipeline 63.03 Tune: dev2010+test2010 ASR outputs test2011 ASR output + SLT pipeline 63.35

  5. Spoken Language Translation SLT pipeline BLEU(MT source) test2010 ASR transcript 70.79 + number conversion 71.37 + punctuation insertion 84.80 + postprocessing 85.17 test2010 ASR out + SLT pipeline 61.82 Punctuation Insertion System BLEU(MT source) Tune: dev2010 ASR transcript test2011 ASR output + SLT pipeline 62.39 Tune: dev2010+test2010 ASR transcripts test2011 ASR output + SLT pipeline 63.03 Tune: dev2010+test2010 ASR outputs test2011 ASR output + SLT pipeline 63.35

  6. Spoken Language Translation SLT pipeline BLEU(MT source) test2010 ASR transcript 70.79 + number conversion 71.37 + punctuation insertion 84.80 + postprocessing 85.17 test2010 ASR out + SLT pipeline 61.82 Punctuation Insertion System BLEU(MT source) Tune: dev2010 ASR transcript test2011 ASR output + SLT pipeline 62.39 Tune: dev2010+test2010 ASR transcripts test2011 ASR output + SLT pipeline 63.03 Tune: dev2010+test2010 ASR outputs test2011 ASR output + SLT pipeline 63.35

  7. Spoken Language Translation SLT pipeline BLEU(MT source) test2010 ASR transcript 70.79 + number conversion 71.37 + punctuation insertion 84.80 + postprocessing 85.17 test2010 ASR out + SLT pipeline 61.82 Punctuation Insertion System BLEU(MT source) Tune: dev2010 ASR transcript test2011 ASR output + SLT pipeline 62.39 Tune: dev2010+test2010 ASR transcripts test2011 ASR output + SLT pipeline 63.03 Tune: dev2010+test2010 ASR outputs test2011 ASR output + SLT pipeline 63.35

  8. Spoken Language Translation SLT pipeline BLEU(MT source) test2010 ASR transcript 70.79 + number conversion 71.37 + punctuation insertion 84.80 + postprocessing 85.17 test2010 ASR out + SLT pipeline 61.82 Punctuation Insertion System BLEU(MT source) Tune: dev2010 ASR transcript test2011 ASR output + SLT pipeline 62.39 Tune: dev2010+test2010 ASR transcripts test2011 ASR output + SLT pipeline 63.03 Tune: dev2010+test2010 ASR outputs test2011 ASR output + SLT pipeline 63.35

  9. Spoken Language Translation SLT pipeline BLEU(MT source) test2010 ASR transcript 70.79 + number conversion 71.37 + punctuation insertion 84.80 + postprocessing 85.17 test2010 ASR out + SLT pipeline 61.82 Punctuation Insertion System BLEU(MT source) Tune: dev2010 ASR transcript test2011 ASR output + SLT pipeline 62.39 Tune: dev2010+test2010 ASR transcripts test2011 ASR output + SLT pipeline 63.03 Tune: dev2010+test2010 ASR outputs test2011 ASR output + SLT pipeline 63.35

  10. Spoken Language Translation SLT pipeline BLEU(MT source) test2010 ASR transcript 70.79 + number conversion 71.37 + punctuation insertion 84.80 + postprocessing 85.17 test2010 ASR out + SLT pipeline 61.82 Punctuation Insertion System BLEU(MT source) Tune: dev2010 ASR transcript test2011 ASR output + SLT pipeline 62.39 Tune: dev2010+test2010 ASR transcripts test2011 ASR output + SLT pipeline 63.03 Tune: dev2010+test2010 ASR outputs test2011 ASR output + SLT pipeline 63.35

  11. Spoken Language Translation SLT pipeline BLEU(MT source) test2010 ASR transcript 70.79 + number conversion 71.37 + punctuation insertion 84.80 + postprocessing 85.17 test2010 ASR out + SLT pipeline 61.82 Punctuation Insertion System BLEU(MT source) Tune: dev2010 ASR transcript test2011 ASR output + SLT pipeline 62.39 Tune: dev2010+test2010 ASR transcripts test2011 ASR output + SLT pipeline 63.03 Tune: dev2010+test2010 ASR outputs test2011 ASR output + SLT pipeline 63.35

  12. Spoken Language Translation SLT pipeline BLEU(MT source) test2010 ASR transcript 70.79 + number conversion 71.37 + punctuation insertion 84.80 + postprocessing 85.17 test2010 ASR out + SLT pipeline 61.82 Punctuation Insertion System BLEU(MT source) Tune: dev2010 ASR transcript test2011 ASR output + SLT pipeline 62.39 Tune: dev2010+test2010 ASR transcripts test2011 ASR output + SLT pipeline 63.03 Tune: dev2010+test2010 ASR outputs test2011 ASR output + SLT pipeline 63.35

  13. Spoken Language Translation SLT pipeline BLEU(MT source) test2010 ASR transcript 70.79 + number conversion 71.37 + punctuation insertion 84.80 + postprocessing 85.17 test2010 ASR out + SLT pipeline 61.82 Punctuation Insertion System BLEU(MT source) Tune: dev2010 ASR transcript test2011 ASR output + SLT pipeline 62.39 Tune: dev2010+test2010 ASR transcripts test2011 ASR output + SLT pipeline 63.03 Tune: dev2010+test2010 ASR outputs test2011 ASR output + SLT pipeline 63.35

  14. Spoken Language Translation SLT pipeline + MT System MT src MT tgt Oracle test2010 ASR transcript 85.17 30.54 33.98 test2010 ASR out UEDIN 61.82 22.89 33.98 test2011 ASR out system0 67.40 27.37 40.44 test2011 ASR out system1 65.73 27.47 40.44 test2011 ASR out system2 65.82 27.48 40.44 test2011 ASR out UEDIN 63.35 26.83 40.44 Table: SLT end-to-end results (BLEU)

  15. Spoken Language Translation SLT pipeline + MT System MT src MT tgt Oracle test2010 ASR transcript 85.17 30.54 33.98 test2010 ASR out UEDIN 61.82 22.89 33.98 test2011 ASR out system0 67.40 27.37 40.44 test2011 ASR out system1 65.73 27.47 40.44 test2011 ASR out system2 65.82 27.48 40.44 test2011 ASR out UEDIN 63.35 26.83 40.44 Table: SLT end-to-end results (BLEU)

  16. Spoken Language Translation SLT pipeline + MT System MT src MT tgt Oracle test2010 ASR transcript 85.17 30.54 33.98 test2010 ASR out UEDIN 61.82 22.89 33.98 test2011 ASR out system0 67.40 27.37 40.44 test2011 ASR out system1 65.73 27.47 40.44 test2011 ASR out system2 65.82 27.48 40.44 test2011 ASR out UEDIN 63.35 26.83 40.44 Table: SLT end-to-end results (BLEU)

  17. Spoken Language Translation SLT pipeline + MT System MT src MT tgt Oracle test2010 ASR transcript 85.17 30.54 33.98 test2010 ASR out UEDIN 61.82 22.89 33.98 test2011 ASR out system0 67.40 27.37 40.44 test2011 ASR out system1 65.73 27.47 40.44 test2011 ASR out system2 65.82 27.48 40.44 test2011 ASR out UEDIN 63.35 26.83 40.44 Table: SLT end-to-end results (BLEU)

  18. Spoken Language Translation SLT pipeline + MT System MT src MT tgt Oracle test2010 ASR transcript 85.17 30.54 33.98 test2010 ASR out UEDIN 61.82 22.89 33.98 test2011 ASR out system0 67.40 27.37 40.44 test2011 ASR out system1 65.73 27.47 40.44 test2011 ASR out system2 65.82 27.48 40.44 test2011 ASR out UEDIN 63.35 26.83 40.44 Table: SLT end-to-end results (BLEU)

  19. Spoken Language Translation SLT pipeline + MT System MT src MT tgt Oracle test2010 ASR transcript 85.17 30.54 33.98 test2010 ASR out UEDIN 61.82 22.89 33.98 test2011 ASR out system0 67.40 27.37 40.44 test2011 ASR out system1 65.73 27.47 40.44 test2011 ASR out system2 65.82 27.48 40.44 test2011 ASR out UEDIN 63.35 26.83 40.44 Table: SLT end-to-end results (BLEU)

  20. Spoken Language Translation SLT pipeline + MT System MT src MT tgt Oracle test2010 ASR transcript 85.17 30.54 33.98 test2010 ASR out UEDIN 61.82 22.89 33.98 test2011 ASR out system0 67.40 27.37 40.44 test2011 ASR out system1 65.73 27.47 40.44 test2011 ASR out system2 65.82 27.48 40.44 test2011 ASR out UEDIN 63.35 26.83 40.44 Table: SLT end-to-end results (BLEU)

  21. Machine Translation Problem • Limited amount of TED talks data, larger amounts of out-of-domain data • Need to make best use of both kinds of data English-French, German-English • Compare approaches to data filtering and PT adaptation (previous work) • Adaptation to TED talks by adding sparse lexicalised features • Explore different tuning setups on in-domain and mixed-domain systems

  22. Machine Translation Problem • Limited amount of TED talks data, larger amounts of out-of-domain data • Need to make best use of both kinds of data English-French, German-English • Compare approaches to data filtering and PT adaptation (previous work) • Adaptation to TED talks by adding sparse lexicalised features • Explore different tuning setups on in-domain and mixed-domain systems

  23. Machine Translation Problem • Limited amount of TED talks data, larger amounts of out-of-domain data • Need to make best use of both kinds of data English-French, German-English • Compare approaches to data filtering and PT adaptation (previous work) • Adaptation to TED talks by adding sparse lexicalised features • Explore different tuning setups on in-domain and mixed-domain systems

  24. Machine Translation Problem • Limited amount of TED talks data, larger amounts of out-of-domain data • Need to make best use of both kinds of data English-French, German-English • Compare approaches to data filtering and PT adaptation (previous work) • Adaptation to TED talks by adding sparse lexicalised features • Explore different tuning setups on in-domain and mixed-domain systems

  25. Machine Translation Problem • Limited amount of TED talks data, larger amounts of out-of-domain data • Need to make best use of both kinds of data English-French, German-English • Compare approaches to data filtering and PT adaptation (previous work) • Adaptation to TED talks by adding sparse lexicalised features • Explore different tuning setups on in-domain and mixed-domain systems

  26. Machine Translation Baseline systems in-domain, mixed domain • Phrase-based/hierarchical Moses • 5gram LMs with modified Kneser-Ney smoothing • German-English: compound splitting [Koehn and Knight, 2003] and syntactic preordering on source side [Collins et al., 2005] Data • Parallel in-domain data: 140K/130K TED talks • Parallel out-of-domain data: Europarl, News Commentary, MultiUN, (10 9 ) • Additional LM data: Gigaword, Newscrawl (fr: 1.3G words, en: 6.4G words) • Dev set: dev2010, Devtest set: test2010, Test set: test2011

  27. Machine Translation Baseline systems System de-en (test2010) IN-PB (CS) 28.26 IN-PB (PRE) 28.04 IN-PB (CS + PRE) 28.54 test2010 System en-fr de-en IN hierarchical 28.94 27.88 IN phrasebased 29.58 28.54 IN+OUT phrasebased 31.67 28.39 + only in-domain LM 30.97 28.61 + gigaword + newscrawl 31.96 30.26

  28. Machine Translation Baseline systems System de-en (test2010) IN-PB (CS) 28.26 IN-PB (PRE) 28.04 IN-PB (CS + PRE) 28.54 test2010 System en-fr de-en IN hierarchical 28.94 27.88 IN phrasebased 29.58 28.54 IN+OUT phrasebased 31.67 28.39 + only in-domain LM 30.97 28.61 + gigaword + newscrawl 31.96 30.26

  29. Machine Translation Baseline systems System de-en (test2010) IN-PB (CS) 28.26 IN-PB (PRE) 28.04 IN-PB (CS + PRE) 28.54 test2010 System en-fr de-en IN hierarchical 28.94 27.88 IN phrasebased 29.58 28.54 IN+OUT phrasebased 31.67 28.39 + only in-domain LM 30.97 28.61 + gigaword + newscrawl 31.96 30.26

  30. Machine Translation Baseline systems System de-en (test2010) IN-PB (CS) 28.26 IN-PB (PRE) 28.04 IN-PB (CS + PRE) 28.54 test2010 System en-fr de-en IN hierarchical 28.94 27.88 IN phrasebased 29.58 28.54 IN+OUT phrasebased 31.67 28.39 + only in-domain LM 30.97 28.61 + gigaword + newscrawl 31.96 30.26

  31. Machine Translation Baseline systems System de-en (test2010) IN-PB (CS) 28.26 IN-PB (PRE) 28.04 IN-PB (CS + PRE) 28.54 test2010 System en-fr de-en IN hierarchical 28.94 27.88 IN phrasebased 29.58 28.54 IN+OUT phrasebased 31.67 28.39 + only in-domain LM 30.97 28.61 + gigaword + newscrawl 31.96 30.26

  32. Machine Translation Baseline systems System de-en (test2010) IN-PB (CS) 28.26 IN-PB (PRE) 28.04 IN-PB (CS + PRE) 28.54 test2010 System en-fr de-en IN hierarchical 28.94 27.88 IN phrasebased 29.58 28.54 IN+OUT phrasebased 31.67 28.39 + only in-domain LM 30.97 28.61 + gigaword + newscrawl 31.96 30.26

  33. Data selection and PT adaptation Bilingual cross-entropy difference [Axelrod et al., 2011] • Select out-of-domain sentences that are similar to in-domain and dissimilar from out-of-domain data • Select 10%, 20%, 50% of OUT data (incl. LM data) In-domain PT + fill-up OUT [Bisazza et al., 2011], [Haddow and Koehn, 2012] • Train phrase-table on both IN and OUT data • Replace all scores of phrase pairs found in IN table with the scores from that table

  34. Data selection and PT adaptation Bilingual cross-entropy difference [Axelrod et al., 2011] • Select out-of-domain sentences that are similar to in-domain and dissimilar from out-of-domain data • Select 10%, 20%, 50% of OUT data (incl. LM data) In-domain PT + fill-up OUT [Bisazza et al., 2011], [Haddow and Koehn, 2012] • Train phrase-table on both IN and OUT data • Replace all scores of phrase pairs found in IN table with the scores from that table

  35. Data selection and PT adaptation test2010 System en-fr de-en IN+OUT 31.67 28.39 IN + 10% OUT 32.30 29.29 + 20% OUT 32.45 29.11 + 50% OUT 32.32 28.68 best + gigaword + newscrawl 32.93 31.06 IN + fill-up OUT 32.19 29.59 + gigaword + newscrawl 32.72 31.30

  36. Data selection and PT adaptation test2010 System en-fr de-en IN+OUT 31.67 28.39 IN + 10% OUT 32.30 29.29 + 20% OUT 32.45 29.11 + 50% OUT 32.32 28.68 best + gigaword + newscrawl 32.93 31.06 IN + fill-up OUT 32.19 29.59 + gigaword + newscrawl 32.72 31.30

  37. Data selection and PT adaptation test2010 System en-fr de-en IN+OUT 31.67 28.39 IN + 10% OUT 32.30 29.29 + 20% OUT 32.45 29.11 + 50% OUT 32.32 28.68 best + gigaword + newscrawl 32.93 31.06 IN + fill-up OUT 32.19 29.59 + gigaword + newscrawl 32.72 31.30

  38. Data selection and PT adaptation test2010 System en-fr de-en IN+OUT 31.67 28.39 IN + 10% OUT 32.30 29.29 + 20% OUT 32.45 29.11 + 50% OUT 32.32 28.68 best + gigaword + newscrawl 32.93 31.06 IN + fill-up OUT 32.19 29.59 + gigaword + newscrawl 32.72 31.30

  39. Sparse feature tuning Adapt to style and vocabulary of TED talks • Add sparse word pair and phrase pair features to in-domain system, tune with online MIRA • Word pairs: indicators of aligned words in source and target • Phrase pairs: depend on phrase segmentation of decoder • Bias translation model towards in-domain style and vocabulary

  40. Sparse feature tuning Adapt to style and vocabulary of TED talks • Add sparse word pair and phrase pair features to in-domain system, tune with online MIRA • Word pairs: indicators of aligned words in source and target • Phrase pairs: depend on phrase segmentation of decoder • Bias translation model towards in-domain style and vocabulary

  41. Sparse feature tuning Adapt to style and vocabulary of TED talks • Add sparse word pair and phrase pair features to in-domain system, tune with online MIRA • Word pairs: indicators of aligned words in source and target • Phrase pairs: depend on phrase segmentation of decoder • Bias translation model towards in-domain style and vocabulary

  42. Sparse feature tuning schemes IN IN OUT training training in-domain mixed-domain model model jackknife tuning direct tuning retuning direct tuning core weights core weights core weights core weights + + + + sparse sparse sparse meta-feature feature feature feature weight weights weights weights

  43. Sparse feature tuning schemes IN IN OUT training training in-domain mixed-domain model model jackknife tuning direct tuning retuning direct tuning core weights core weights core weights core weights + + + + sparse sparse sparse meta-feature feature feature feature weight weights weights weights

  44. Direct tuning with MIRA • Tune on development set • Online MIRA: Select hope/fear translations from a 30best list • Sentence-level BLEU scores • Separate learning rate for core features to reduce fluctuation and keep MIRA training more stable • Learning rate set to 0.1 for core features (1.0 for sparse features)

  45. Direct tuning with MIRA • Tune on development set • Online MIRA: Select hope/fear translations from a 30best list • Sentence-level BLEU scores • Separate learning rate for core features to reduce fluctuation and keep MIRA training more stable • Learning rate set to 0.1 for core features (1.0 for sparse features)

  46. Direct tuning with MIRA • Tune on development set • Online MIRA: Select hope/fear translations from a 30best list • Sentence-level BLEU scores • Separate learning rate for core features to reduce fluctuation and keep MIRA training more stable • Learning rate set to 0.1 for core features (1.0 for sparse features)

  47. Direct tuning with MIRA • Tune on development set • Online MIRA: Select hope/fear translations from a 30best list • Sentence-level BLEU scores • Separate learning rate for core features to reduce fluctuation and keep MIRA training more stable • Learning rate set to 0.1 for core features (1.0 for sparse features)

  48. Direct tuning with MIRA Sparse feature sets Source sentence: [a language] [is a] [flash of] [the human spirit] [.] Hypothesis translation: [une langue] [est une] [flash de] [l’ esprit humain] [.] Word pair features Phrase pair features wp a ∼ une=2 pp a,language ∼ une,langue=1 wp language ∼ langue=1 pp is,a ∼ est,une=1 wp is ∼ est=1 pp flash,of ∼ flash,de=1 wp flash ∼ flash=1 . . . wp of ∼ de=1 . . .

  49. Direct tuning with MIRA Sparse feature sets Source sentence: [a language] [is a] [flash of] [the human spirit] [.] Hypothesis translation: [une langue] [est une] [flash de] [l’ esprit humain] [.] Word pair features Phrase pair features wp a ∼ une=2 pp a,language ∼ une,langue=1 wp language ∼ langue=1 pp is,a ∼ est,une=1 wp is ∼ est=1 pp flash,of ∼ flash,de=1 wp flash ∼ flash=1 . . . wp of ∼ de=1 . . .

  50. Direct tuning with MIRA Sparse feature sets Source sentence: [a language] [is a] [flash of] [the human spirit] [.] Hypothesis translation: [une langue] [est une] [flash de] [l’ esprit humain] [.] Word pair features Phrase pair features wp a ∼ une=2 pp a,language ∼ une,langue=1 wp language ∼ langue=1 pp is,a ∼ est,une=1 wp is ∼ est=1 pp flash,of ∼ flash,de=1 wp flash ∼ flash=1 . . . wp of ∼ de=1 . . .

  51. Direct tuning with MIRA Sparse feature sets Source sentence: [a language] [is a] [flash of] [the human spirit] [.] Hypothesis translation: [une langue] [est une] [flash de] [l’ esprit humain] [.] Word pair features Phrase pair features wp a ∼ une=2 pp a,language ∼ une,langue=1 wp language ∼ langue=1 pp is,a ∼ est,une=1 wp is ∼ est=1 pp flash,of ∼ flash,de=1 wp flash ∼ flash=1 . . . wp of ∼ de=1 . . .

  52. Sparse feature tuning schemes IN IN OUT training training in-domain mixed-domain model model jackknife tuning direct tuning retuning direct tuning core weights core weights core weights core weights + + + + sparse sparse sparse meta-feature feature feature feature weight weights weights weights

  53. Jackknife tuning with MIRA • To avoid overfitting to tuning set, train lexicalised features on all in-domain training data fold 1 nbest 1 fold 10 nbest 10 • Train 10 systems on MT system 1 in-domain data, leaving out fold 1 one fold at a time MT system 2 fold 2 • Then translate each fold fold 3 MT system .. fold .. with respective system MT system 9 fold 9 • Iterative parameter mixing fold 10 MT system 10 by running MIRA on all 10 systems in parallel

  54. Jackknife tuning with MIRA • To avoid overfitting to tuning set, train lexicalised features on all in-domain training data fold 1 nbest 1 fold 10 nbest 10 • Train 10 systems on MT system 1 in-domain data, leaving out fold 1 one fold at a time MT system 2 fold 2 • Then translate each fold fold 3 MT system .. fold .. with respective system MT system 9 fold 9 • Iterative parameter mixing fold 10 MT system 10 by running MIRA on all 10 systems in parallel

  55. Jackknife tuning with MIRA • To avoid overfitting to tuning set, train lexicalised features on all in-domain training data fold 1 nbest 1 fold 10 nbest 10 • Train 10 systems on MT system 1 in-domain data, leaving out fold 1 one fold at a time MT system 2 fold 2 • Then translate each fold fold 3 MT system .. fold .. with respective system MT system 9 fold 9 • Iterative parameter mixing fold 10 MT system 10 by running MIRA on all 10 systems in parallel

  56. Sparse feature tuning schemes IN IN OUT training training in-domain mixed-domain model model direct tuning jackknife tuning retuning direct tuning core weights core weights core weights core weights + + + + sparse sparse sparse meta-feature feature feature feature weight weights weights weights

  57. Retuning with MIRA Motivation • Tuning sparse features for large translation models is time/memory-consuming • Avoid overhead of jackknife tuning on larger data sets • Port tuned features from in-domain to mixed-domain models Feature integration • Rescale jackknife-tuned features to integrate into mixed-domain model • Combine into aggregated meta-feature with a single weight • During decoding, meta-feature weight is applied to all sparse features of the same class • Retuning step: core weights of mixed-domain model tuned together with meta-feature weight

  58. Retuning with MIRA Motivation • Tuning sparse features for large translation models is time/memory-consuming • Avoid overhead of jackknife tuning on larger data sets • Port tuned features from in-domain to mixed-domain models Feature integration • Rescale jackknife-tuned features to integrate into mixed-domain model • Combine into aggregated meta-feature with a single weight • During decoding, meta-feature weight is applied to all sparse features of the same class • Retuning step: core weights of mixed-domain model tuned together with meta-feature weight

  59. Retuning with MIRA Motivation • Tuning sparse features for large translation models is time/memory-consuming • Avoid overhead of jackknife tuning on larger data sets • Port tuned features from in-domain to mixed-domain models Feature integration • Rescale jackknife-tuned features to integrate into mixed-domain model • Combine into aggregated meta-feature with a single weight • During decoding, meta-feature weight is applied to all sparse features of the same class • Retuning step: core weights of mixed-domain model tuned together with meta-feature weight

  60. Retuning with MIRA Motivation • Tuning sparse features for large translation models is time/memory-consuming • Avoid overhead of jackknife tuning on larger data sets • Port tuned features from in-domain to mixed-domain models Feature integration • Rescale jackknife-tuned features to integrate into mixed-domain model • Combine into aggregated meta-feature with a single weight • During decoding, meta-feature weight is applied to all sparse features of the same class • Retuning step: core weights of mixed-domain model tuned together with meta-feature weight

  61. Retuning with MIRA Motivation • Tuning sparse features for large translation models is time/memory-consuming • Avoid overhead of jackknife tuning on larger data sets • Port tuned features from in-domain to mixed-domain models Feature integration • Rescale jackknife-tuned features to integrate into mixed-domain model • Combine into aggregated meta-feature with a single weight • During decoding, meta-feature weight is applied to all sparse features of the same class • Retuning step: core weights of mixed-domain model tuned together with meta-feature weight

  62. Retuning with MIRA Motivation • Tuning sparse features for large translation models is time/memory-consuming • Avoid overhead of jackknife tuning on larger data sets • Port tuned features from in-domain to mixed-domain models Feature integration • Rescale jackknife-tuned features to integrate into mixed-domain model • Combine into aggregated meta-feature with a single weight • During decoding, meta-feature weight is applied to all sparse features of the same class • Retuning step: core weights of mixed-domain model tuned together with meta-feature weight

  63. Retuning with MIRA Motivation • Tuning sparse features for large translation models is time/memory-consuming • Avoid overhead of jackknife tuning on larger data sets • Port tuned features from in-domain to mixed-domain models Feature integration • Rescale jackknife-tuned features to integrate into mixed-domain model • Combine into aggregated meta-feature with a single weight • During decoding, meta-feature weight is applied to all sparse features of the same class • Retuning step: core weights of mixed-domain model tuned together with meta-feature weight

  64. Results with sparse features test2010 System en-fr de-en IN, MERT 29.58 28.54 IN, MIRA 30.28 28.31 + word pairs 30.36 28.45 + phrase pairs 30.62 28.40 + word pairs (JK) 30.80 28.78 + phrase pairs (JK) 30.77 28.61 Table: Direct tuning and jackknife tuning on in-domain data • en-fr: +0.34/+0.52 BLEU with direct/jackknife tuning • de-en: +0.14/+0.47 BLEU with direct/jackknife tuning

  65. Results with sparse features test2010 System en-fr de-en IN, MERT 29.58 28.54 IN, MIRA 30.28 28.31 + word pairs 30.36 28.45 + phrase pairs 30.62 28.40 + word pairs (JK) 30.80 28.78 + phrase pairs (JK) 30.77 28.61 Table: Direct tuning and jackknife tuning on in-domain data • en-fr: +0.34/+0.52 BLEU with direct/jackknife tuning • de-en: +0.14/+0.47 BLEU with direct/jackknife tuning

  66. Results with sparse features test2010 System en-fr de-en IN, MERT 29.58 28.54 IN, MIRA 30.28 28.31 + word pairs 30.36 28.45 + phrase pairs 30.62 28.40 + word pairs (JK) 30.80 28.78 + phrase pairs (JK) 30.77 28.61 Table: Direct tuning and jackknife tuning on in-domain data • en-fr: +0.34/+0.52 BLEU with direct/jackknife tuning • de-en: +0.14/+0.47 BLEU with direct/jackknife tuning

  67. MT Results en-fr de-en System test2010 test2011 test2010 test2011 IN + %OUT, MIRA 33.22 40.02 28.90 34.03 + word pairs 33.59 39.95 28.93 33.88 + phrase pairs 33.44 40.02 29.13 33.99 IN + %OUT, MERT 32.32 39.36 29.13 33.29 + retune(word pair JK) 32.90 40.31 29.58 33.31 + retune(phrase pairs JK) 32.69 39.32 29.38 33.23 Submission system (grey) + gigaword + newscrawl 33.98 40.44 31.28 36.03 Table: (Data selection + Sparse features (direct/retuning)) + large LMs

  68. MT Results en-fr de-en System test2010 test2011 test2010 test2011 IN + %OUT, MIRA 33.22 40.02 28.90 34.03 + word pairs 33.59 39.95 28.93 33.88 + phrase pairs 33.44 40.02 29.13 33.99 IN + %OUT, MERT 32.32 39.36 29.13 33.29 + retune(word pair JK) 32.90 40.31 29.58 33.31 + retune(phrase pairs JK) 32.69 39.32 29.38 33.23 Submission system (grey) + gigaword + newscrawl 33.98 40.44 31.28 36.03 Table: (Data selection + Sparse features (direct/retuning)) + large LMs

  69. MT Results en-fr de-en System test2010 test2011 test2010 test2011 IN + %OUT, MIRA 33.22 40.02 28.90 34.03 + word pairs 33.59 39.95 28.93 33.88 + phrase pairs 33.44 40.02 29.13 33.99 IN + %OUT, MERT 32.32 39.36 29.13 33.29 + retune(word pair JK) 32.90 40.31 29.58 33.31 + retune(phrase pairs JK) 32.69 39.32 29.38 33.23 Submission system (grey) + gigaword + newscrawl 33.98 40.44 31.28 36.03 Table: (Data selection + Sparse features (direct/retuning)) + large LMs

  70. MT Results en-fr de-en System test2010 test2011 test2010 test2011 IN + %OUT, MIRA 33.22 40.02 28.90 34.03 + word pairs 33.59 39.95 28.93 33.88 + phrase pairs 33.44 40.02 29.13 33.99 IN + %OUT, MERT 32.32 39.36 29.13 33.29 + retune(word pair JK) 32.90 40.31 29.58 33.31 + retune(phrase pairs JK) 32.69 39.32 29.38 33.23 Submission system (grey) + gigaword + newscrawl 33.98 40.44 31.28 36.03 Table: (Data selection + Sparse features (direct/retuning)) + large LMs

  71. MT Results en-fr de-en System test2010 test2011 test2010 test2011 IN + %OUT, MIRA 33.22 40.02 28.90 34.03 + word pairs 33.59 39.95 28.93 33.88 + phrase pairs 33.44 40.02 29.13 33.99 IN + %OUT, MERT 32.32 39.36 29.13 33.29 + retune(word pair JK) 32.90 40.31 29.58 33.31 + retune(phrase pairs JK) 32.69 39.32 29.38 33.23 Submission system (grey) + gigaword + newscrawl 33.98 40.44 31.28 36.03 Table: (Data selection + Sparse features (direct/retuning)) + large LMs

  72. MT Results en-fr de-en System test2010 test2011 test2010 test2011 IN + %OUT, MIRA 33.22 40.02 28.90 34.03 + word pairs 33.59 39.95 28.93 33.88 + phrase pairs 33.44 40.02 29.13 33.99 IN + %OUT, MERT 32.32 39.36 29.13 33.29 + retune(word pairs JK) 32.90 40.31 29.58 33.31 + retune(phrase pairs JK) 32.69 39.32 29.38 33.23 Submission system (grey) + gigaword + newscrawl 33.98 40.44 31.28 36.03 Table: (Data selection + Sparse features (direct/retuning)) + large LMs

  73. Summary MT • Used data selection for final systems (IN+OUT) • Sparse lexicalised features to adapt to style and vocabulary of TED talks, larger gains with jackknife tuning • Compared three tuning setups for sparse features • On test2010, all systems with sparse features improved over baselines, less systematic differences on test2011 • Best system for de-en: test2010: IN+10%OUT, MERT+retune(wp JK) test2011: IN+10%OUT, MIRA • Best systems for en-fr: test2010: IN+20%OUT, MIRA+wp test2011: IN+20%OUT, MERT+retune(wp JK)

  74. Summary MT • Used data selection for final systems (IN+OUT) • Sparse lexicalised features to adapt to style and vocabulary of TED talks, larger gains with jackknife tuning • Compared three tuning setups for sparse features • On test2010, all systems with sparse features improved over baselines, less systematic differences on test2011 • Best system for de-en: test2010: IN+10%OUT, MERT+retune(wp JK) test2011: IN+10%OUT, MIRA • Best systems for en-fr: test2010: IN+20%OUT, MIRA+wp test2011: IN+20%OUT, MERT+retune(wp JK)

  75. Summary MT • Used data selection for final systems (IN+OUT) • Sparse lexicalised features to adapt to style and vocabulary of TED talks, larger gains with jackknife tuning • Compared three tuning setups for sparse features • On test2010, all systems with sparse features improved over baselines, less systematic differences on test2011 • Best system for de-en: test2010: IN+10%OUT, MERT+retune(wp JK) test2011: IN+10%OUT, MIRA • Best systems for en-fr: test2010: IN+20%OUT, MIRA+wp test2011: IN+20%OUT, MERT+retune(wp JK)

  76. Summary MT • Used data selection for final systems (IN+OUT) • Sparse lexicalised features to adapt to style and vocabulary of TED talks, larger gains with jackknife tuning • Compared three tuning setups for sparse features • On test2010, all systems with sparse features improved over baselines, less systematic differences on test2011 • Best system for de-en: test2010: IN+10%OUT, MERT+retune(wp JK) test2011: IN+10%OUT, MIRA • Best systems for en-fr: test2010: IN+20%OUT, MIRA+wp test2011: IN+20%OUT, MERT+retune(wp JK)

  77. Summary MT • Used data selection for final systems (IN+OUT) • Sparse lexicalised features to adapt to style and vocabulary of TED talks, larger gains with jackknife tuning • Compared three tuning setups for sparse features • On test2010, all systems with sparse features improved over baselines, less systematic differences on test2011 • Best system for de-en: test2010: IN+10%OUT, MERT+retune(wp JK) test2011: IN+10%OUT, MIRA • Best systems for en-fr: test2010: IN+20%OUT, MIRA+wp test2011: IN+20%OUT, MERT+retune(wp JK)

  78. Summary MT • Used data selection for final systems (IN+OUT) • Sparse lexicalised features to adapt to style and vocabulary of TED talks, larger gains with jackknife tuning • Compared three tuning setups for sparse features • On test2010, all systems with sparse features improved over baselines, less systematic differences on test2011 • Best system for de-en: test2010: IN+10%OUT, MERT+retune(wp JK) test2011: IN+10%OUT, MIRA • Best systems for en-fr: test2010: IN+20%OUT, MIRA+wp test2011: IN+20%OUT, MERT+retune(wp JK)

Recommend

FBK's Machine Translation Systems for IWSLT 2012's TED Lectures Nick Ruiz, Arianna Bisazza

FBK's Machine Translation Systems for IWSLT 2012's TED Lectures Nick Ruiz, Arianna Bisazza

FBK's Machine Translation Systems for IWSLT 2012's TED Lectures Nick Ruiz, Arianna Bisazza Roldano Cattoni, Marcello Federico FBK's Machine Translation Systems for IWSLT 2012's TED Lectures 1 Hong Kong, 6 December 2012 2 Outline Common

310 views • 27 slides
KIT EN-FR Systems for the IWSLT 2012 Mohammed Mediani, Yuqi Zhang, Thanh-Le Ha, Jan Niehues, Eunah

KIT EN-FR Systems for the IWSLT 2012 Mohammed Mediani, Yuqi Zhang, Thanh-Le Ha, Jan Niehues, Eunah

KIT EN-FR Systems for the IWSLT 2012 Mohammed Mediani, Yuqi Zhang, Thanh-Le Ha, Jan Niehues, Eunah Cho, Teresa Herrmann, Rainer Krgel and Alex Waibel KIT University of the State of Baden-Wuerttemberg and www.kit.edu National Research

490 views • 23 slides
The HIT-LTRC Machine Translation System for IWSLT 2012 Xiaoning Zhu, Yiming Cui, Conghui Zhu,

The HIT-LTRC Machine Translation System for IWSLT 2012 Xiaoning Zhu, Yiming Cui, Conghui Zhu,

The HIT-LTRC Machine Translation System for IWSLT 2012 Xiaoning Zhu, Yiming Cui, Conghui Zhu, Tiejun Zhao and Hailong Cao Harbin Institute of Technology Outline Introduction System summary Pialign Experiments

433 views • 14 slides
NAISTs Machine Translation Systems for IWSLT 2020 Conversational Speech Translation Task Ryo

NAISTs Machine Translation Systems for IWSLT 2020 Conversational Speech Translation Task Ryo

NAISTs Machine Translation Systems for IWSLT 2020 Conversational Speech Translation Task Ryo Fukuda 1 , Katsuhito Sudoh 1 , and Satoshi Nakamura 1,2 1 Nara Institute of Science and Technology 2 AIP Center, RIKEN, Japan Brief Overview Ch

389 views • 23 slides
Evaluations: 2009-2012 Molly Hageboeck and Micah Frumkin Management Systems International

Evaluations: 2009-2012 Molly Hageboeck and Micah Frumkin Management Systems International

Meta-Evaluation of USAID Evaluations: 2009-2012 Molly Hageboeck and Micah Frumkin Management Systems International November 25, 2013 USAID Meta-Evaluation Context Early Evaluation Leader 1970 - 1994 1970s -- Developed/adopted the

477 views • 25 slides
A Statistical Evaluation of the Decathlon Scoring Systems 2011-2012 DE RIJDT Annelin One of the

A Statistical Evaluation of the Decathlon Scoring Systems 2011-2012 DE RIJDT Annelin One of the

A Statistical Evaluation of the Decathlon Scoring Systems 2011-2012 DE RIJDT Annelin One of the most challenging disciplines in sports??? End of the decathlon at the Olympic Games in Beijing (2008) Challenges to develop a scoring system?

435 views • 30 slides
Evaluation DEMMS: Evaluation of Multimedia What are the Evaluation lectures about: When

Evaluation DEMMS: Evaluation of Multimedia What are the Evaluation lectures about: When

Evaluation DEMMS: Evaluation of Multimedia What are the Evaluation lectures about: When to evaluate Systems What kinds of evaluation are possible Predictive evaluations Robert Villa Traditional user experiments

551 views • 10 slides
Question-Answering: Evaluation, Systems, Resources Ling573 NLP Systems & Applications

Question-Answering: Evaluation, Systems, Resources Ling573 NLP Systems & Applications

Question-Answering: Evaluation, Systems, Resources Ling573 NLP Systems & Applications April 5, 2011 Roadmap Rounding dimensions of QA Evaluation, TREC QA systems: Alternate Approaches ISIs Webclopedia

676 views • 56 slides
Workload-Driven Architectural Evaluation Evaluation in Uniprocessors Decisions made only after

Workload-Driven Architectural Evaluation Evaluation in Uniprocessors Decisions made only after

Workload-Driven Architectural Evaluation Evaluation in Uniprocessors Decisions made only after quantitative evaluation For existing systems: comparison and procurement evaluation For future systems: careful extrapolation from known quantities

889 views • 76 slides
Interim Evaluation and Long-Term Evaluation Planning August 23, 2012 Agenda 2 Introductions

Interim Evaluation and Long-Term Evaluation Planning August 23, 2012 Agenda 2 Introductions

County of Santa Clara Community Corrections Partnership 1 AB 109 Evaluation: Interim Evaluation and Long-Term Evaluation Planning August 23, 2012 Agenda 2 Introductions AB 109 Interim Evaluation Overview Project Timeline CCP

88 views • 8 slides
Evaluation & Systems Ling573 Systems & Applications April 7, 2016 Roadmap

Evaluation & Systems Ling573 Systems & Applications April 7, 2016 Roadmap

Evaluation & Systems Ling573 Systems & Applications April 7, 2016 Roadmap Evaluation: Scoring without models Content selection: Unsupervised word-weighting approaches Non-trivial baseline system example:

515 views • 28 slides
Evaluation of Recommender Systems Radek Pel anek Summary Proper evaluation is important, but

Evaluation of Recommender Systems Radek Pel anek Summary Proper evaluation is important, but

Evaluation of Recommender Systems Radek Pel anek Summary Proper evaluation is important, but really difficult. Evaluation: Typical Questions Do recommendations work? Do they increase sales? How much? Which algorithm should we prefer for

623 views • 59 slides
Performance Evaluation of Performance Evaluation of Phasor Measurement Systems Phasor

Performance Evaluation of Performance Evaluation of Phasor Measurement Systems Phasor

PNNL-SA-60867 Panel: International Experience in PMU Applications Performance Evaluation of Performance Evaluation of Phasor Measurement Systems Phasor Measurement Systems Henry Huang (PNNL), Bogdan Kasztenny (GE), Vahid Madani (PG&E), Ken

452 views • 29 slides
Monter un syst` eme de traduction automatique statistique bas e sur les s equences de mots:

Monter un syst` eme de traduction automatique statistique bas e sur les s equences de mots:

Experimenting with Phrase-Based Statistical Translation 1 Monter un syst` eme de traduction automatique statistique bas e sur les s equences de mots: Le cas de la campagne d evaluation IWSLT Philippe Langlais RALI/DIRO

681 views • 42 slides
Millburn School District 24 Evaluation Plan Formal Evaluation Process October 2012 Agenda

Millburn School District 24 Evaluation Plan Formal Evaluation Process October 2012 Agenda

Millburn School District 24 Evaluation Plan Formal Evaluation Process October 2012 Agenda Background Information Components of District 24 Evaluation System Observations - Formal, Informal Forms Needed Whats Next?

388 views • 18 slides
Evaluation 101 Energy Efficiency Program Evaluation By Nick Hall TecMarket Works February 8,

Evaluation 101 Energy Efficiency Program Evaluation By Nick Hall TecMarket Works February 8,

Delaware Webinar Evaluation 101 Energy Efficiency Program Evaluation By Nick Hall TecMarket Works February 8, 2012 Delaware Evaluation Webinar February 2012 Page 1 of 54 Workshop Objectives Create a greater understanding of evaluation,

840 views • 54 slides
Evaluation of the anthropogenic Evaluation of the anthropogenic impact on surface water systems:

Evaluation of the anthropogenic Evaluation of the anthropogenic impact on surface water systems:

Purpose of the study Evaluation of the anthropogenic Evaluation of the anthropogenic impact on surface water systems: impact on surface water systems: Evaluate the degree of contamination with sewage case of Lower Arges Basin, case of

184 views • 3 slides
Test and Evaluation of Localization and Tracking Systems Nader Moayeri NIST Presented at 3rd

Test and Evaluation of Localization and Tracking Systems Nader Moayeri NIST Presented at 3rd

Test and Evaluation of Localization and Tracking Systems Nader Moayeri NIST Presented at 3rd Invitational Workshop on Opportunistic RF Localization for Next Generation Wireless Devices May 7, 2012 Introduction Lack of standardized Test

514 views • 18 slides
Performance Evaluation of Queuing Systems Introduction to Queuing Systems System Performance

Performance Evaluation of Queuing Systems Introduction to Queuing Systems System Performance

Performance Evaluation of Queuing Systems Introduction to Queuing Systems System Performance Measures & Littles Law Equilibrium Solution of Birth-Death Processes Analysis of Single-Station Queuing Systems Dr Conor McArdle EE414 -

469 views • 33 slides
Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on

Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on

Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] 1 Telematics 2 / Performance Evaluation (WS 17/18): 11 Systematic Evaluation of Complex Systems Systematic Evaluation of Complex Systems

566 views • 40 slides
Summarization Evaluation & Systems Ling573 Systems and Applications April 4, 2017 Roadmap

Summarization Evaluation & Systems Ling573 Systems and Applications April 4, 2017 Roadmap

Summarization Evaluation & Systems Ling573 Systems and Applications April 4, 2017 Roadmap Summarization evaluation: Intrinsic: Model-based: ROUGE, Pyramid Model-free Content selection Model classes

383 views • 27 slides
Evaluation of resource Evaluation of resource arbitration methods for arbitration methods for

Evaluation of resource Evaluation of resource arbitration methods for arbitration methods for

Computer Science 12 Design Automation for Embedded Systems Evaluation of resource Evaluation of resource arbitration methods for arbitration methods for multi-core real-time systems multi-core real-time systems Paper presentation at WCET

554 views • 23 slides
Advanced Database Management Systems Query Processing: Query Evaluation Alvaro A A Fernandes

Advanced Database Management Systems Query Processing: Query Evaluation Alvaro A A Fernandes

Advanced Database Management Systems Query Processing: Query Evaluation Alvaro A A Fernandes School of Computer Science, University of Manchester AAAF (School of CS, Manchester) Advanced DBMSs 2012-2013 1 / 111 Outline Algorithmic

1.23k views • 111 slides
RWTH Aachen Machine Translation System: {Arabic, Chinese, German}-English MT Track Stephan Peitz,

RWTH Aachen Machine Translation System: {Arabic, Chinese, German}-English MT Track Stephan Peitz,

RWTH Aachen Machine Translation System: {Arabic, Chinese, German}-English MT Track Stephan Peitz, Markus Freitag, Saab Mansour, Minwei Feng, Joern Wuebker surname@cs.rwth-aachen.de IWSLT 2012, Hongkong December 6, 2012 Human Language

374 views • 19 slides

More recommend