Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation Melvin Johnson∗, Mike Schuster∗, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, Jeffrey Dean Google {melvinp,schuster}@google.com Abstract We propose a simple solution to use a single Neural Machine Translation (NMT) model to translate between multiple languages. Our solu- tion requires no changes to the model architec- ture from a standard NMT system but instead introduces an artificial token at the beginning of the input sentence to specify the required target language. Using a shared wordpiece vo- cabulary, our approach enables Multilingual NMT systems using a single model. On the WMT’14 benchmarks, a single multilingual model achieves comparable performance for English→French and surpasses state-of-the- art results for English→German. Similarly, a single multilingual model surpasses state- of-the-art results for French→English and German→English on WMT’14 and WMT’15 benchmarks, respectively. On production cor- pora, multilingual models of up to twelve language pairs allow for better translation of many individual pairs. Our models can also learn to perform implicit bridging between lan- guage pairs never seen explicitly during train- ing, showing that transfer learning and zero- shot translation is possible for neural transla- tion. Finally, we show analyses that hints at a universal interlingua representation in our mod- els and also show some interesting examples when mixing languages. 1 Introduction End-to-end Neural Machine Translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015; Cho et al., 2014) is an approach to machine ∗Corresponding authors. translation that has rapidly gained adoption in many large-scale settings (Zhou et al., 2016; Wu et al., 2016; Crego and et al., 2016). Almost all such systems are built for a single language pair — so far there has not been a sufficiently simple and efficient way to handle multiple language pairs using a single model without making significant changes to the basic NMT architecture. In this paper we introduce a simple method to translate between multiple languages using a single model, taking advantage of multilingual data to im- prove NMT for all languages involved. Our method requires no change to the traditional NMT model architecture. Instead, we add an artificial token to the input sequence to indicate the required target lan- guage, a simple amendment to the data only. All other parts of the system — encoder, decoder, atten- tion, and shared wordpiece vocabulary as described in Wu et al., (2016) — stay exactly the same. This method has several attractive benefits: • Simplicity: Since no changes are made to the architecture of the model, scaling to more lan- guages is trivial — any new data is simply added, possibly with over- or under-sampling such that all languages are appropriately rep- resented, and used with a new token if the tar- get language changes. Since no changes are made to the training procedure, the mini-batches for training are just sampled from the overall mixed-language training data just like for the single-language case. Since no a-priori deci- sions about how to …

Content truncated to 500 words