Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 898–909, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 898 Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention Biao Zhang1 Ivan Titov1,2 Rico Sennrich3,1 1School of Informatics, University of Edinburgh 2ILLC, University of Amsterdam 3Institute of Computational Linguistics, University of Zurich B.Zhang@ed.ac.uk, ititov@inf.ed.ac.uk, sennrich@cl.uzh.ch Abstract The general trend in NLP is towards in- creasing model capacity and performance via deeper neural networks. However, simply stacking more layers of the popular Trans- former architecture for machine translation re- sults in poor convergence and high computa- tional overhead. Our empirical analysis sug- gests that convergence is poor due to gradi- ent vanishing caused by the interaction be- tween residual connections and layer normal- ization. We propose depth-scaled initialization (DS-Init), which decreases parameter variance at the initialization stage, and reduces out- put variance of residual connections so as to ease gradient back-propagation through nor- malization layers. To address computational cost, we propose a merged attention sublayer (MAtt) which combines a simplified average- based self-attention sublayer and the encoder- decoder attention sublayer on the decoder side. Results on WMT and IWSLT translation tasks with five translation directions show that deep Transformers with DS-Init and MAtt can sub- stantially outperform their base counterpart in terms of BLEU (+1.1 BLEU on average for 12-layer models), while matching the decod- ing speed of the baseline model thanks to the efficiency improvements of MAtt.1 1 Introduction The capability of deep neural models of handling complex dependencies has benefited various ar- tificial intelligence tasks, such as image recogni- tion where test error was reduced by scaling VGG nets (Simonyan and Zisserman, 2015) up to hun- dreds of convolutional layers (He et al., 2015). In NLP, deep self-attention networks have enabled large-scale pretrained language models such as BERT (Devlin et al., 2019) and GPT (Radford 1Source code for reproduction is available at https:// github.com/bzhangGo/zero 0 1 2 3 4 5 6 Gradient Norm (a) Encoder Transformer 6L Transformer 12L Transformer 18L Transformer+DS-Init 6L Transformer+DS-Init 12L Transformer+DS-Init 18L 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Layer Depth 0 1 2 3 4 5 6 Gradient Norm (b) Decoder Transformer 6L Transformer 12L Transformer 18L Transformer+DS-Init 6L Transformer+DS-Init 12L Transformer+DS-Init 18L Figure 1: Gradient norm (y-axis) of each encoder layer (top) and decoder layer (bottom) in Transformer with respect to layer depth (x-axis). Gradients are estimated with ∼3k target tokens at the beginning of training. “DS-Init”: the proposed depth-scaled initialization. “6L”: 6 layers. Solid lines indi- cate the vanilla Transformer, and dashed lines denote our pro- posed method. During back-propagation, gradients in Trans- former gradually vanish from high layers to low layers. et al., 2018) to boost state-of-the-art (SOTA) per- formance on downstream applications. By con- trast, though neural machine translation (NMT) gained encouraging improvement when shifting from a shallow architecture (Bahdanau et al., …

Content truncated to 500 words