Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder

Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder Thanh-Le Ha, Jan Niehues, Alex Waibel Institute for Anthropomatics and Robotics KIT - Karlsruhe Institute of Technology, Germany firstname.lastname@kit.edu Abstract In this paper, we present our first attempts in building a multi- lingual Neural Machine Translation framework under a uni- fied approach in which the information shared among lan- guages can be helpful in the translation of individual lan- guage pairs. We are then able to employ attention-based Neural Machine Translation for many-to-many multilingual translation tasks. Our approach does not require any spe- cial treatment on the network architecture and it allows us to learn minimal number of free parameters in a standard way of training. Our approach has shown its effectiveness in an under-resourced translation scenario with considerable im- provements up to 2.6 BLEU points. In addition, we point out a novel way to make use of monolingual data with Neural Machine Translation using the same approach with a 3.15- BLEU-score gain in IWSLT’16 English→German transla- tion task. 1. Introduction Neural Machine Translation (NMT) has shown its effective- ness in translation tasks when NMT systems perform best in recent machine translation campaigns [1, 2]. Compared to phrase-based Statistical Machine Translation (SMT) which is basically an ensemble of different features trained and tuned separately, NMT directly modeling the translation relation- ship between source and target sentences. Unlike SMT, NMT does not require large monolingual data to achieve good per- formances. An NMT system consists of an encoder which recur- sively reads and represents the whole source sentence into a context vector and a recurrent decoder which takes the con- text vector and its previous state to predict the next target word. It is then trained in an end-to-end fashion to learn parameters which maximizes the likelihood between the out- puts and the references. Recently, attention-based NMT has been featured in most state-of-the-art systems. First intro- duced by [3], attention mechanism is integrated in decoder side as feedforward layers. It allows the NMT to decide which source words should take part in the prediction pro- cess of the next target words. It helps to improve NMTs significantly. Nevertheless, since the attention mechanism is specific to a particular source sentence and the considering target word, it is also specific to particular language pairs. Some recent work has focused on extending the NMT framework to multilingual scenarios. By training such net- work using parallel corpora in number of different languages, NMT could benefit from additional information embedded in a common semantic space across languages. Basically, the proposed NMT are required to employ multiple encoders or multiple decoders to deal with multilinguality. Furthermore, in order to avoid the tight dependency of the attention mech- anism to specific language pairs, they also need to modify their architecture to combine either the encoders or the atten- tion layers. These modifications are specific to the purpose of the tasks as well. Thus, those multilingual NMTs are more complicated, much more free parameters to learn and more difficult …

Content truncated to 500 words