Learning Deep Transformer Models for Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810–1822 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1810 Learning Deep Transformer Models for Machine Translation Qiang Wang1, Bei Li1, Tong Xiao1,2∗, Jingbo Zhu1,2, Changliang Li3, Derek F. Wong4, Lidia S. Chao4 1NLP Lab, Northeastern University, Shenyang, China 2NiuTrans Co., Ltd., Shenyang, China 3Kingsoft AI Lab, Beijing, China 4NLP2CT Lab, University of Macau, Macau, China wangqiangneu@gmail.com, libei neu@outlook.com, {xiaotong,zhujingbo}@mail.neu.edu.com, lichangliang@kingsoft.com, {derekfw,lidiasc}@um.edu.mo Abstract Transformer is the state-of-the-art model in recent machine translation evaluations. Two strands of research are promising to im- prove models of this kind: the first uses wide networks (a.k.a. Transformer-Big) and has been the de facto standard for the de- velopment of the Transformer system, and the other uses deeper language representation but faces the difficulty arising from learn- ing deep networks. Here, we continue the line of research on the latter. We claim that a truly deep Transformer model can surpass the Transformer-Big counterpart by 1) proper use of layer normalization and 2) a novel way of passing the combination of previous layers to the next. On WMT’16 English- German, NIST OpenMT’12 Chinese-English and larger WMT’18 Chinese-English tasks, our deep system (30/25-layer encoder) out- performs the shallow Transformer-Big/Base baseline (6-layer encoder) by 0.4∼2.4 BLEU points. As another bonus, the deep model is 1.6X smaller in size and 3X faster in training than Transformer-Big1. 1 Introduction Neural machine translation (NMT) models have advanced the previous state-of-the-art by learn- ing mappings between sequences via neural net- works and attention mechanisms (Sutskever et al., 2014; Bahdanau et al., 2015). The earliest of these read and generate word sequences using a series of recurrent neural network (RNN) units, and the improvement continues when 4-8 layers are stacked for a deeper model (Luong et al., 2015; Wu et al., 2016). More recently, the system based on multi-layer self-attention (call it Trans- former) has shown strong results on several large- ∗Corresponding author. 1The source code is available at https://github. com/wangqiangneu/dlcl scale tasks (Vaswani et al., 2017). In particu- lar, approaches of this kind benefit greatly from a wide network with more hidden states (a.k.a. Transformer-Big), whereas simply deepening the network has not been found to outperform the “shallow” counterpart (Bapna et al., 2018). Do deep models help Transformer? It is still an open question for the discipline. For vanilla Transformer, learning deeper net- works is not easy because there is already a rel- atively deep model in use2. It is well known that such deep networks are difficult to optimize due to the gradient vanishing/exploding problem (Pas- canu et al., 2013; Bapna et al., 2018). We note that, despite the significant development effort, simply stacking more layers cannot benefit the system and leads to a disaster of training in some of our exper- iments. A promising attempt to address this issue is Bapna et al. (2018)’s work. They trained a 16- layer Transformer encoder by using an enhanced attention model. In this work, we …

Content truncated to 500 words