Parameter Sharing Methods for Multilingual Self-Attentional Translation Models

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 261–271 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64027 Parameter Sharing Methods for Multilingual Self-Attentional Translation Models Devendra Singh Sachan Data Solutions Team Petuum Inc. Pittsburgh, USA devendra.singh@petuum.com Graham Neubig Language Technologies Institute Carnegie Mellon University Pittsburgh, USA gneubig@cs.cmu.edu Abstract In multilingual neural machine translation, it has been shown that sharing a single trans- lation model between multiple languages can achieve competitive performance, sometimes even leading to performance gains over bilin- gually trained models. However, these im- provements are not uniform; often multilin- gual parameter sharing results in a decrease in accuracy due to translation models not be- ing able to accommodate different languages in their limited parameter space. In this work, we examine parameter sharing techniques that strike a happy medium between full sharing and individual training, specifically focusing on the self-attentional Transformer model. We find that the full parameter sharing approach leads to increases in BLEU scores mainly when the target languages are from a similar language family. However, even in the case where target languages are from different fam- ilies where full parameter sharing leads to a noticeable drop in BLEU scores, our proposed methods for partial sharing of parameters can lead to substantial improvements in translation accuracy.1 1 Introduction Neural machine translation (NMT; Sutskever et al. (2014); Cho et al. (2014)) is now the de-facto stan- dard in MT research due to its relative simplicity of implementation, ability to perform end-to-end training, and high translation accuracy. Early ap- proaches to NMT used recurrent neural networks (RNNs), usually LSTMs (Hochreiter and Schmid- huber, 1997), in their encoder and decoder layers, with the addition of an attention mechanism (Bah- danau et al., 2014; Luong et al., 2015) to focus more on specific encoded source words when de- ciding the next translation target output. Recently, 1Data and code of this paper is available at: https://github.com/DevSinghSachan/multilingual_nmt Shared Encoder Decoder 2 Source Language: "En" Target Language 2: "Nl" Decoder 1 Target Language 1: "De" (a) Shared encoder, separate decoder (Dong et al., 2015). Shared Encoder Source Language: "En" Target Language 2: "Nl" Shared Decoder Target Language 1: "De" (b) Shared encoder and decoder (Johnson et al., 2017). Shared Encoder Decoder 2 Source Language: "En" Target Language 2: "Nl" Target Language 1: "De" Decoder 1 Shareable Parameters (c) Proposed shared decoder with partial parameter sharing. Figure 1: Examples of MTL frameworks for the translation of one source language (for example “En”) to two target languages (for example “De”, “Nl”). The principle remains the same with more than two target languages. Best viewed in color. the NMT research community has been transition- ing from RNNs to an alternative method for encod- ing sentences using self-attention (Vaswani et al., 2017), represented by the so-called “Transformer” model, which both improves the speed of process- ing sentences on computational hardware such as GPUs due to its lack of recurrence, and achieves impressive results. In parallel to …

Content truncated to 500 words