Neural Translation with Pytorch GTC 2017 JEREMY HOWARD - PowerPoint PPT Presentation

Aug 14, 2022 •209 likes •450 views

Neural Translation with Pytorch GTC 2017 JEREMY HOWARD @JEREMYPHOWARD Im assuming some knowledge of Python Jupyter Numpy Word RNNs vectors Some review today course.fast.ai https://github.com/ jph00/part2 Our destination

Neural Translation with Pytorch GTC 2017 JEREMY HOWARD @JEREMYPHOWARD
I’m assuming some knowledge of… Python Jupyter Numpy Word RNNs vectors Some review today
course.fast.ai
https://github.com/ jph00/part2
Our destination https://github.com/ jph00/part2
Data source Created by Chris Callison-Burch Crawled millions of web pages Used 'a set of simple heuristics’ • Transform French URLs onto English URLs • i.e. replacing "fr" with "en" and about 40 other hand-written rules Assume that these documents are translations of each other
The dataset – just the questions
Tokenizing Because we are translating at word level, we need to tokenize the text first. There are many tokenizers available, but we found we got best results using these simple heuristics.
Final preprocessing result
Unrolled stacked RNNs for sequences Input  Hidden Hidden  Output Hidden  Hidden word 3 input word 2 input Output Hidden Input word 1 input
Equivalent recursive diagram Repeat for 1  n-1 char n input Repeat for 1  n-1 Initialize to zeros Initialize to zeros
This and following 3 slides thanks to Chris Manning (Stanford) https://simons.berkeley.edu/talks/christopher-manning-2017-3-27
* Equation from: “Grammar as a Foreign Language”
Beam search Quelle est la population du Canada ? Que le en Quoi les pour What is canada 's population ?
Neural Translation with Pytorch GTC 2017 JEREMY HOWARD @JEREMYPHOWARD

Recommend

Neural Machine Translation Gongbo Tang 8 October 2018 Outline Neural Machine Translation 1

Neural Machine Translation Gongbo Tang 8 October 2018 Outline Neural Machine Translation 1 Advances and Challenges 2 Gongbo Tang Neural Machine Translation 2/52 Neural Machine Translation Figure Recurrent neural network based NMT

906 views • 73 slides

Neural Machine Translation Philipp Koehn 6 October 2020 Philipp Koehn Machine Translation:

Neural Machine Translation Philipp Koehn 6 October 2020 Philipp Koehn Machine Translation: Neural Machine Translation 6 October 2020 Language Models 1 Modeling variants feed-forward neural network recurrent neural network long

485 views • 36 slides

Convolutional over Recurrent Encoder for Neural Machine Translation Praveen Dakwale and Christof

Annual Conference of the European Association for Machine Translation 2017 Convolutional over Recurrent Encoder for Neural Machine Translation Praveen Dakwale and Christof Monz Neural Machine Translation End to end neural network with RNN

354 views • 20 slides

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of Edinburgh R. Sennrich MT 2018 12 1 / 27 Todays Lecture So far, main focus of lecture was on: neural machine translation research

740 views • 46 slides

Adaptive Multi-pass Decoder for Neural Machine Translation EMNLP 2018

Adaptive Multi-pass Decoder for Neural Machine Translation EMNLP 2018 http://aclweb.org/anthology/D18-1048 Neural Machine Translation(NMT) The encoder-decoder are widely used in neural machine translation the encoder transforms the source

121 views • 11 slides

Googles Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Googles Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation Author: Melvin Johnson , Mike Schuster , Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vigas, Martin Wattenberg, Greg

344 views • 16 slides

Introduction to Neural Machine Translation Gongbo Tang 16 September 2019 Outline Why Neural

Introduction to Neural Machine Translation Gongbo Tang 16 September 2019 Outline Why Neural Machine Translation ? 1 Introduction to Neural Networks 2 Neural Language Models 3 Gongbo Tang Introduction to Neural Machine Translation 2/38

686 views • 50 slides

Neural machine translation with less supervision CMSC 470 Marine Carpuat Neural MT only helps

Neural machine translation with less supervision CMSC 470 Marine Carpuat Neural MT only helps in high-resource settings Ongoing research Learn from other sources of supervision than pairs (E,F) Monolingual text Multiple languages

479 views • 29 slides

Neural Machine Translation: Breaking the Performance Plateau Rico Sennrich Institute for

Neural Machine Translation: Breaking the Performance Plateau Rico Sennrich Institute for Language, Cognition and Computation University of Edinburgh July 4 2016 Rico Sennrich Neural Machine Translation 1 / 15 Is Machine Translation Getting

452 views • 21 slides

Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu,

Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu 1 Machine Translation Automated translation using computer software 2 Machine Translation Rule-based

650 views • 26 slides

BREAKTHROUGHS IN NEURAL MACHINE TRANSLATION Olof Mogren Chalmers University of Technology

BREAKTHROUGHS IN NEURAL MACHINE TRANSLATION Olof Mogren Chalmers University of Technology 2016-09-29 COMING SEMINARS T oday: Olof Mogren Neural Machine Translation October 6: John Wiedenhoeft Fast Bayesian inference in Hidden Markov

607 views • 22 slides

Inferring Translation Candidates for Multilingual Dictionary Generation with Multi-Way Neural

Inferring Translation Candidates for Multilingual Dictionary Generation with Multi-Way Neural Machine Translation Mihael Arcan, Daniel Torregrosa*, Sina Ahmadi* and John P. McCrae This publication has emanated from research supported in part

591 views • 42 slides

What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured Prediction? Graham Neubig @ ICLR Workshop on Deep Reinforcement Learning Meets Structured Prediction 5/6/2019 Types of Prediction Types of

1.73k views • 127 slides

8 Neural MT 2: Attentional Neural MT In the past chapter, we described a simple model for neural

8 Neural MT 2: Attentional Neural MT In the past chapter, we described a simple model for neural machine translation, which uses an encoder to encode sentences as a fixed-length vector. However, in some ways, this view is overly simplified, and

635 views • 9 slides

An Introduction to Neural Machine Translation Prof. John D. Kelleher @johndkelleher ADAPT Centre

An Introduction to Neural Machine Translation Prof. John D. Kelleher @johndkelleher ADAPT Centre for Digital Content Technology Dublin Institute of Technology, Ireland June 25, 2018 1 / 57 Outline The Neural Machine Translation Revolution

611 views • 57 slides

What can Statistical Machine Translation teach Neural Text Generation about Optimization? Graham

What can Statistical Machine Translation teach Neural Text Generation about Optimization? Graham Neubig @ NAACL Workshop on Methods for Optimizing and Evaluating Neural Language Generation 6/6/2019 or How to Optimize your Neural Generation

638 views • 34 slides

Neural Machine Translation: directions for improvement CMSC 470 Marine Carpuat How can we

Neural Machine Translation: directions for improvement CMSC 470 Marine Carpuat How can we improve on state-of-the-art machine translation approaches? Model Training Data Objective Algorithm Addressing domain mismatch

490 views • 36 slides

Investigations on Phrase-based Decoding with Recurrent Neural Network Language and Translation

Investigations on Phrase-based Decoding with Recurrent Neural Network Language and Translation Models Tamer Alkhouli, Felix Rietig, and Hermann Ney surname@cs.rwth-aachen.de Tenth Workshop on Statistical Machine Translation Lisbon, Portugal

563 views • 24 slides

Sparse and Constrained Attention for Neural Machine Translation Chaitanya Malaviya 1 ,

Sparse and Constrained Attention for Neural Machine Translation Chaitanya Malaviya 1 , Pedro Ferreira 2 , Andr F.T. Martins 2,3 1 Carnegie Mellon University, 2 Instituto Superior Tcnico, 3 Unbabel 1 Adequacy in Neural Machine

574 views • 22 slides

Effective Approaches to Attention-based Neural Machine Translation Thang Luong Hieu Pham and Chris

Effective Approaches to Attention-based Neural Machine Translation Thang Luong Hieu Pham and Chris Manning EMNLP 2015 Presented by: Yunan Zhang Neural Machine Translation Attention Mechanism (Sutskever et al., 2014) (Bahdanau et al., 2015) _ suis

499 views • 34 slides

Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT 2015 Graham

Neural Reranking Improves Subjective Quality of Machine Translation Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT 2015 Graham Neubig, Makoto Morishita, Satoshi Nakamura Nara Institute of Science and

713 views • 20 slides

Neural Machine Translation Decoding Philipp Koehn 8 October 2020 Philipp Koehn Machine

Neural Machine Translation Decoding Philipp Koehn 8 October 2020 Philipp Koehn Machine Translation: Neural Machine Translation Decoding 8 October 2020 Inference 1 Given a trained model ... we now want to translate test sentences We

682 views • 39 slides

Syntactic Translation Lattices Felix Stahlberg, Adria de Gispert, Eva Hasler, and Bill Byrne

Neural Machine Translation by Minimising the Bayes-risk with Respect to Syntactic Translation Lattices Felix Stahlberg, Adria de Gispert, Eva Hasler, and Bill Byrne Department of Engineering Neural Machine Translation by Minimising the

1.97k views • 12 slides

Document Context Neural Machine Translation with Memory Networks Sameen Maruf, Gholamreza Haffari

Document Context Neural Machine Translation with Memory Networks Document Context Neural Machine Translation with Memory Networks Sameen Maruf, Gholamreza Haffari Faculty of Information Technology Monash University July 17, 2017 1 / 30

901 views • 78 slides