A neural interlingua for multilingual machine translation

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 84–92 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64009 A neural interlingua for multilingual machine translation Yichao Lu∗, Phillip Keung∗, Faisal Ladhak, Vikas Bhardwaj, Shaonan Zhang, Jason Sun {yichaolu,keung,faisall,vikab,shaonanz,jasun}@amazon.com Amazon Inc. Abstract We incorporate an explicit neural interlin- gua into a multilingual encoder-decoder neural machine translation (NMT) architecture. We demonstrate that our model learns a language- independent representation by performing di- rect zero-shot translation (without using pivot translation), and by using the source sentence embeddings to create an English Yelp review classifier that, through the mediation of the neural interlingua, can also classify French and German reviews. Furthermore, we show that, despite using a smaller number of pa- rameters than a pairwise collection of bilingual NMT models, our approach produces compa- rable BLEU scores for each language pair in WMT15. 1 Introduction 1.1 Multilingual Machine Translation Neural machine translation (NMT) relies on word and sentence embeddings to encode the seman- tic information needed for translation. The stan- dard attentional encoder-decoder models (Bah- danau et al., 2015) for bilingual NMT decom- pose naturally into separate encoder and decoder subnetworks for the source and target languages. This factorization has inspired various forms of multilingual NMT models that extended the orig- inal bilingual framework to handle more language pairs simultaneously. We refer to NMT models that accept sentences from one source language and produce outputs in one target language as ‘bilingual’. We contrast this with ‘multilingual’ NMT models, which support more than one source and/or target languages within the same model. The naive approach to multilingual machine translation would train a model for each language pair, which scales quadratically with the number ∗ Equal contribution of languages in the corpus. Instead, by com- bining language-specific encoders and decoders in different ways, Dong et al. (2015), Zoph and Knight (2016), Luong et al. (2016), and Firat et al. (2016a) have explored the one source-to- many target, many source-to-one target, and many source-to-many target multilingual MT settings. The multi-way shared attention model (Firat et al., 2016a) is closest to our work, in that they consider the large-scale, many-to-many scenario with mul- tiple encoders and decoders. It is also possible to adapt existing bilingual NMT models to the many-to-many case without changing the architecture at all. The universal encoder-decoder approach (Ha et al., 2016; John- son et al., 2017) constructs a shared vocabulary for all languages in the dataset, and use just one encoder and decoder for multilingual translation. In addition, Johnson et al. (2017) introduce direct zero-shot translation, which refers to the task of translating between language pairs without paral- lel text or pivoting through an intermediate lan- guage like English. Direct zero-shot translation may yield lower BLEU scores than pivot-based approaches, but avoids doubling the latency and computational overhead (due to translating the source sentence twice,) which is a concern for large-scale, productionized MT systems. Nonetheless, both the multi-way shared at- tention model and …

Content truncated to 500 words