Multi-layer Representation Fusion for Neural Machine Translation

Proceedings of the 27th International Conference on Computational Linguistics, pages 3015–3026 Santa Fe, New Mexico, USA, August 20-26, 2018. 3015 Multi-layer Representation Fusion for Neural Machine Translation Qiang Wang†, Fuxue Li†,‡, Tong Xiao†∗, Yanyang Li†, Yinqiao Li†, Jingbo Zhu† †Natural Language Processing Lab., Northeastern University ‡YingKou Institute of Technology wangqiangneu@gmail.com, lifuxue119@163.com xiaotong@mail.neu.edu.cn, blamedrlee@outlook.com li.yin.qiao.2012@hotmail.com, zhujingbo@mail.neu.edu.cn Abstract Neural machine translation systems require a number of stacked layers for deep models. But the prediction depends on the sentence representation of the top-most layer with no access to low-level representations. This makes it more difficult to train the model and poses a risk of information loss to prediction. In this paper, we propose a multi-layer representation fusion (MLRF) approach to fusing stacked layers. In particular, we design three fusion functions to learn a better representation from the stack. Experimental results show that our approach yields improvements of 0.92 and 0.56 BLEU points over the strong Transformer baseline on IWSLT German-English and NIST Chinese-English MT tasks respectively. The result is new state-of- the-art in German-English translation. 1 Introduction Neural models that use the encoder-decoder architecture to capture the translation equivalence relation between languages have been widely adopted over the last few years. The simplest of these relies on one recurrent neural network layer on both the encoder and decoder sides (Bahdanau et al., 2015), whereas others have successfully explored the high-level representation of language via deeper models (Wu et al., 2016; Gehring et al., 2017b; Vaswani et al., 2017). It has been noted that increasing the network depth is one of the factors contributing to the success of neural machine translation (NMT). To this end, one can stack a number of layers for an enriched sentence representation. E.g., popular NMT systems require 4 stacked layers or more for state-of-the-art results on large-scale translation tasks (Wu et al., 2016). Unfortunately, a straightforward implementation of deep neural networks has been found to underperform shallow models due to the poor convergence. One solution to this problem is to add identity mapping (or shortcut connection) between layers. A popular method is the residual network (He et al., 2016a), which has been used in several successful systems, e.g., GNMT and Transformer. This model offers a good way to ease the information flow in the stack. But they do not make full use of the multi-level representations of the deep networks. E.g., word predictions depend on the sentence representation of the top-most layer with no access to low-level representations. Recent evidence sup- ports that it helps when features from lower-level layers are involved in prediction (Xiong et al., 2017; Gehring et al., 2017b). A good instance is DenseNet that applies a fully-connected convolutional net- work to computer vision (Huang et al., 2017a). But it is still rare to see studies on this issue in machine translation. In this paper, we take a further step along the line of this research. Drawing on previous successful attempts to create shortcut connections between adjacent layers, we propose a multi-layer representation fusion (MLRF) approach …

Content truncated to 500 words