Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism

Proceedings of NAACL-HLT 2016, pages 866–875, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism Orhan Firat Middle East Technical University orhan.firat@ceng.metu.edu.tr Kyunghyun Cho New York University Yoshua Bengio University of Montreal CIFAR Senior Fellow Abstract We propose multi-way, multilingual neural machine translation. The proposed approach enables a single neural translation model to translate between multiple languages, with a number of parameters that grows only lin- early with the number of languages. This is made possible by having a single atten- tion mechanism that is shared across all lan- guage pairs. We train the proposed multi- way, multilingual model on ten language pairs from WMT’15 simultaneously and observe clear performance improvements over models trained on only one language pair. In partic- ular, we observe that the proposed model sig- nificantly improves the translation quality of low-resource language pairs. 1 Introduction Neural Machine Translation It has been shown that a deep (recurrent) neural network can success- fully learn a complex mapping between variable- length input and output sequences on its own. Some of the earlier successes in this task have, for in- stance, been handwriting recognition (Bottou et al., 1997; Graves et al., 2009) and speech recogni- tion (Graves et al., 2006; Chorowski et al., 2015). More recently, a general framework of encoder- decoder networks has been found to be effective at learning this kind of sequence-to-sequence mapping by using two recurrent neural networks (Cho et al., 2014b; Sutskever et al., 2014). A basic encoder-decoder network consists of two recurrent networks. The first network, called an en- coder, maps an input sequence of variable length into a point in a continuous vector space, resulting in a fixed-dimensional context vector. The other re- current neural network, called a decoder, then gener- ates a target sequence again of variable length start- ing from the context vector. This approach however has been found to be inefficient in (Cho et al., 2014a) when handling long sentences, due to the difficulty in learning a complex mapping between an arbitrary long sentence and a single fixed-dimensional vector. In (Bahdanau et al., 2014), a remedy to this issue was proposed by incorporating an attention mecha- nism to the basic encoder-decoder network. The at- tention mechanism in the encoder-decoder network frees the network from having to map a sequence of arbitrary length to a single, fixed-dimensional vec- tor. Since this attention mechanism was introduced to the encoder-decoder network for machine trans- lation, neural machine translation, which is purely based on neural networks to perform full end-to-end translation, has become competitive with the exist- ing phrase-based statistical machine translation in many language pairs (Jean et al., 2015; Gulcehre et al., 2015; Luong et al., 2015b). Multilingual Neural Machine Translation Ex- isting machine translation systems, mostly based on a phrase-based system or its variants, work by di- rectly mapping a symbol or a subsequence of sym- bols in a source language to its corresponding sym- bol or subsequence in …

Content truncated to 500 words