Syntactically Guided Neural Machine Translation Felix Stahlberg, - PowerPoint PPT Presentation

Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne Department of Engineering Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

Neural machine translation (NMT) vs. Hiero NMT Hiero • Simple beam search* • Searches over a vast number of translations • No explicit coverage • CKY parses cover the mechanism* complete source sentence • Limited vocabulary size* • Very large vocabularies, open to extension • Long-range context (RNN) • Limited LM context, weak translation model *: Vanilla formulation of attentional NMT according Bahdanau et al., 2015 Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

Combining NMT and Hiero scores x : Source sentence • NMT left-to-right factorization: 𝑈 : Target sentence y = 𝑧 1 • NMT+Hiero via log-linear model combination Hiero predictive posteriors through FST weight pushing UNK score is used for NMT OOVs Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

FST weight pushing Hiero lattice: <s>|0.5 </s>|0.1 b|0.5 a|0.4 a|0.8 c|0.4 </s>|1.0 b|0.5 b|0.8 c|0.9 Hiero lattice after weight pushing: <s>|0.409 b|0.061 </s>|1.0 a|0.042 a|0.042 a|0.939 c|0.208 c|0.208 </s>|1.0 b|1.0 b|0.75 b|0.75 c|1.0 𝑄 𝐼𝑗𝑓𝑠𝑝 𝑧 3 = a s a, 𝐲 = 0.042 𝑄 𝐼𝑗𝑓𝑠𝑝 𝑧 3 = c s a, 𝐲 = 0.208 𝑄 𝐼𝑗𝑓𝑠𝑝 𝑧 3 = b s a, 𝐲 = 0.75 Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

Results on news-test2014 English-German English-French (BLEU) (BLEU) Baselines and related work Hiero baseline (de Gispert et al., 2010) 19.44 32.86 Basic NMT (RNNsearch) (Bahdanau et al., 2015) 16.31 30.42 RNNsearch-LV + UNK Replace (Jean et al., 2015) 19.40 34.60 This work Syntactically guided NMT ( 𝜇 𝐼𝑗𝑓𝑠𝑝 = 0 ) 20.69 35.37 Syntactically guided NMT (tuned 𝜇 𝑂𝑁𝑈 , 𝜇 𝐼𝑗𝑓𝑠𝑝 ) 21.87 36.61 Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

Results on news-test2015 (English-German) Search space # of node expansions BLEU per sentence 100-best rescoring 2,233.6 22.9 (Depth-First Search: 832.1) 1000-best rescoring 21,686.2 23.5 (Depth-First Search: 6,221.8) Lattice-based 244.3 24.0 (Syntactically guided NMT) NMT baseline: 19.5 BLEU Hiero baseline (with NPLM): 21.7 BLEU Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

Conclusion • Using syntactic SMT to guide neural machine translation yields great potential • Our lattice-based approach is faster and better than n-best list rescoring • More discussion in the paper • NMT modelling vs. search errors • Local softmax • Beam size • Lattice size • … Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

References • Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR • Adria de Gispert, Gonzalo Iglesias, Graeme Blackwood, Eduardo R Banga, and William Byrne. 2010. Hierarchical phrase-based translation with weighted finite-state transducers and shallow-n grammars. Computational Linguistics, 36(3):505 – 533. • Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015a. On using very large target vocabulary for neural machine translation. In ACL, pages 1 – 10. Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

Thanks Code available at http://ucam-smt.github.io/sgnmt/html Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

BACKUP Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

Results Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

Beam size Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

Lattice size Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

Data Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

RNN Update Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne