Semi-supervised Learning for Neural Machine Translation Yong Cheng - PowerPoint PPT Presentation

Jul 17, 2023 •371 likes •650 views

Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu 1 Machine Translation Automated translation using computer software 2 Machine Translation Rule-based

Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu 1
Machine Translation Automated translation using computer software 2
Machine Translation Rule-based Machine Translation 1970s Example-based Machine Translation 1984 Statical Machine Translation (SMT) 1993 Neural Machine Translation � NMT � 2014 Trends: learning to translate from DATA 3
Machine Translation Parallel corpora are usually limited in & & quantity quality coverage Monolingual Corpora Parallel Corpora 4
Monolingual Corpora Used in SMT and NMT N-gram language model in SMT Koehn et al., [2007] Monolingual corpora as decipherment Ravi and Knight [2011] Integrate a neural language model into NMT. Gulccehre et al. [2015] Additional pseudo parallel corpus. Sennrich et al. [2016] 5
Supervised Training Parallel Corpus Objective 6
Unsupervised Training Monolingual Corpus 7
cc Our Approach — Autoencoders x bushi yu shalong juxing le huitan 8
cc Our Approach — Autoencoders ! θ ) P ( y | x ; x bushi yu shalong juxing le huitan 9
cc Our Approach — Autoencoders y latent Bush held a talk with sharon ! θ ) P ( y | x ; x bushi yu shalong juxing le huitan 10
cc Our Approach — Autoencoders ! θ ) P ( x | y ; y latent Bush held a talk with sharon ! θ ) P ( y | x ; x bushi yu shalong juxing le huitan 11
cc Our Approach — Autoencoders ′ x bushi yu shalong juxing le huitan ! θ ) P ( x | y ; y latent Bush held a talk with sharon ! θ ) P ( y | x ; x bushi yu shalong juxing le huitan 12
cc Our Approach — Autoencoders source autoencoder target autoencoder 13
Unsupervised Training (Autoencoders) Monolingual Corpus target autoencoder 14
Semi-supervised Training Training Objective 15
Translation Results Compared with Moses (SMT) and RNNSearch (NMT) 16
Translation Results Compared with Moses (SMT) and RNNSearch (NMT) 17
Translation Results Compared with Moses (SMT) and RNNSearch (NMT) 18
Translation Results Compared with Moses (SMT) and RNNSearch (NMT) 19
Translation Results Compared with Moses (SMT) and RNNSearch (NMT) 20
Translation Results Compared with Sennrich et al. [2015a] 21
Example Translation of Monolingual Corpus 22
Conclusion Monolingual corpora is an important resource for neural machine translation. We have proposed a semi-supervised approach to training bidirectional neural machine translation models for exploiting monolingual corpora. As our method is sensitive to the OOVs present in monolingual corpora, we plan to integrate Jean et al. (2015)’s technique on using very large vocabulary into our approach. 23
Thank You ! 24
Effect of Sample Size ZH-EN EN-ZH 25
Effect of OOV ratio ZH-EN EN-ZH 26

Recommend

Semi-Supervised Learning Maria-Florina Balcan 03/30/2015 Readings: Semi-Supervised Learning.

Semi-Supervised Learning Maria-Florina Balcan 03/30/2015 Readings: Semi-Supervised Learning. Encyclopedia of Machine Learning. Jerry Zhu, 2010 Combining Labeled and Unlabeled Data with Co- Training. Avrim Blum, Tom Mitchell. COLT

953 views • 45 slides

Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site

CS11-747 Neural Networks for NLP Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site https://phontron.com/class/nn4nlp2018/ Supervised, Unsupervised, Semi-supervised Most models handled here are supervised learning

670 views • 32 slides

Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site

CS11-747 Neural Networks for NLP Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Supervised, Unsupervised, Semi-supervised Most models handled here are supervised learning

580 views • 39 slides

Semi-Supervised Learning and Text Analysis Machine Learning 10-701 November 29, 2005 Tom M.

Semi-Supervised Learning and Text Analysis Machine Learning 10-701 November 29, 2005 Tom M. Mitchell Carnegie Mellon University Document Classification: Bag of Words Approach aardvark 0 about 2 all 2 Africa 1 apple 0 anxious 0 ...

747 views • 53 slides

NEURAL RENDERING MODEL (NRM): JOINT GENERATION AND PREDICTION FOR SEMI-SUPERVISED LEARNING Tan

NEURAL RENDERING MODEL (NRM): JOINT GENERATION AND PREDICTION FOR SEMI-SUPERVISED LEARNING Tan Nguyen Joint work with Nhat Ho, Ankit B. Patel, Anima Anandkumar , Michael I. Jordan, and Richard G. Baraniuk Deep Learning The Current

621 views • 25 slides

Neural Machine Translation Gongbo Tang 8 October 2018 Outline Neural Machine Translation 1

Neural Machine Translation Gongbo Tang 8 October 2018 Outline Neural Machine Translation 1 Advances and Challenges 2 Gongbo Tang Neural Machine Translation 2/52 Neural Machine Translation Figure Recurrent neural network based NMT

906 views • 73 slides

Fundamentals of Machine Learning for Neural Machine Translation Dr. John D. Kelleher ADAPT

Fundamentals of Machine Learning for Neural Machine Translation Dr. John D. Kelleher ADAPT Centre for Digital Content Technology Dublin Institute of Technology, Ireland 1 Introduction This paper 1 presents a short introduction to neural

321 views • 15 slides

What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured Prediction? Graham Neubig @ ICLR Workshop on Deep Reinforcement Learning Meets Structured Prediction 5/6/2019 Types of Prediction Types of

1.73k views • 127 slides

IN5550: Neural Methods in Natural Language Processing Lecture 2 Supervised Machine Learning:

IN5550: Neural Methods in Natural Language Processing Lecture 2 Supervised Machine Learning: from Linear Models to Neural Networks Andrey Kutuzov, Vinit Ravishankar, Lilja vrelid, Stephan Oepen, & Erik Velldal University of Oslo 24

457 views • 42 slides

IN5550: Neural Methods in Natural Language Processing Lecture 2 Supervised Machine Learning:

IN5550: Neural Methods in Natural Language Processing Lecture 2 Supervised Machine Learning: from Linear Models to Neural Networks Andrey Kutuzov, Vinit Ravishankar, Jeremy Barnes, Lilja vrelid, Stephan Oepen, & Erik Velldal University

1.95k views • 191 slides

Neural Machine Translation Philipp Koehn 6 October 2020 Philipp Koehn Machine Translation:

Neural Machine Translation Philipp Koehn 6 October 2020 Philipp Koehn Machine Translation: Neural Machine Translation 6 October 2020 Language Models 1 Modeling variants feed-forward neural network recurrent neural network long

485 views • 36 slides

Convolutional over Recurrent Encoder for Neural Machine Translation Praveen Dakwale and Christof

Annual Conference of the European Association for Machine Translation 2017 Convolutional over Recurrent Encoder for Neural Machine Translation Praveen Dakwale and Christof Monz Neural Machine Translation End to end neural network with RNN

354 views • 20 slides

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of Edinburgh R. Sennrich MT 2018 12 1 / 27 Todays Lecture So far, main focus of lecture was on: neural machine translation research

740 views • 46 slides

Adaptive Multi-pass Decoder for Neural Machine Translation EMNLP 2018

Adaptive Multi-pass Decoder for Neural Machine Translation EMNLP 2018 http://aclweb.org/anthology/D18-1048 Neural Machine Translation(NMT) The encoder-decoder are widely used in neural machine translation the encoder transforms the source

121 views • 11 slides

Tangent-Normal Adversarial Regularization for Semi-Supervised Learning Bing Yu , Jingfeng Wu

Tangent-Normal Adversarial Regularization for Semi-Supervised Learning Bing Yu , Jingfeng Wu , Jinwen Ma, Zhanxing Zhu Peking University Beijing Institute of Big Data Research June, 2019 Semi-supervised learning (SSL) Suppose we

201 views • 18 slides

CPSC 340: Machine Learning and Data Mining Non-Parametric Models Summer 2020 Course Map

CPSC 340: Machine Learning and Data Mining Non-Parametric Models Summer 2020 Course Map Machine Learning Approaches Supervised Semi-supervised Unsupervised Reinforcement Learning Learning Learning Learning Classification Regression

1.06k views • 57 slides

Coupled Semi-Supervised Learning for Information Extraction Andrew Carlson, Justin Betteridge,

Coupled Semi-Supervised Learning for Information Extraction Andrew Carlson, Justin Betteridge, Richard C. Wang, Estevam R. Hruschka Jr. and Tom M. Mitchell Machine Learning Department Carnegie Mellon University February 4, 2010 Friday,

405 views • 24 slides

Author Identification Using Semi-supervised Learning Ioannis Kourtis and Efstathios Stamatatos

Author Identification Using Semi-supervised Learning Ioannis Kourtis and Efstathios Stamatatos University of the Aegean Outline Introduction The proposed method Common n-grams SVM Semi-supervised learning Evaluation

533 views • 17 slides

Support Vector Machines (SVMs). Semi-Supervised Learning. Semi-Supervised SVMs.

Support Vector Machines (SVMs). Semi-Supervised Learning. Semi-Supervised SVMs. Maria-Florina Balcan 03/25/2015 Support Vector Machines (SVMs). One of the most theoretically well motivated and practically most e ff ective

792 views • 40 slides

Parallelizing Semi- ReDAS Lab Supervised Learning Algorithms with MapReduce Nick Gauthier

Parallelizing Semi- ReDAS Lab Supervised Learning Algorithms with MapReduce Nick Gauthier Advisor: Dr. Rakesh Verma COMPUTER SCIENCE Motivation Semi-supervised learning(SSL) algorithms are slow when working with large-scale data.

801 views • 14 slides

Strong Baselines for Neural Semi-supervised Learning under Domain Shift Sebastian Ruder Barbara

Strong Baselines for Neural Semi-supervised Learning under Domain Shift Sebastian Ruder Barbara Plank Learning under Domain Shift 2 Learning under Domain Shift State-of-the-art domain adaptation approaches 2 Learning under Domain Shift

1.37k views • 112 slides

Machine Translation using Deep Learning Methods Max Fomin Michael Zolotov Sequence to Sequence

Machine Translation using Deep Learning Methods Max Fomin Michael Zolotov Sequence to Sequence Learning with Neural Networks Learning Phrase Representations using RNN Encoder Decoder for Statistical Machine Translation Topics

909 views • 41 slides

Beyond Weight Tying: Learning Joint Input-Output Embeddings for Neural Machine Translation

1 Beyond Weight Tying: Learning Joint Input-Output Embeddings for Neural Machine Translation Nikolaos Pappas 1 , Lesly Miculicich 1 , 2 , James Henderson 1 1 Idiap Research Institute, Switzerland 2 Ecole polytechnique f ed erale de

456 views • 18 slides

Neural Machine Translation: Breaking the Performance Plateau Rico Sennrich Institute for

Neural Machine Translation: Breaking the Performance Plateau Rico Sennrich Institute for Language, Cognition and Computation University of Edinburgh July 4 2016 Rico Sennrich Neural Machine Translation 1 / 15 Is Machine Translation Getting

452 views • 21 slides