Rethinking Skip Connection with Layer Normalization

Proceedings of the 28th International Conference on Computational Linguistics, pages 3586–3598 Barcelona, Spain (Online), December 8-13, 2020 3586 Rethinking Skip Connection with Layer Normalization Fenglin Liu1∗, Xuancheng Ren2∗, Zhiyuan Zhang2, Xu Sun2, Yuexian Zou1,3† 1ADSPLAB, School of ECE, Peking University, Shenzhen, China 2MOE Key Laboratory of Computational Linguistics, School of EECS, Peking University 3Peng Cheng Laboratory, Shenzhen, China {fenglinliu98, renxc, zzy1210, xusun, zouyx}@pku.edu.cn Abstract Skip connection is a widely-used technique to improve the performance and the convergence of deep neural networks, which is believed to relieve the difficulty in optimization due to non-linearity by propagating a linear component through the neural network layers. However, from another point of view, it can also be seen as a modulating mechanism between the input and the output, with the input scaled by a pre-defined value one. In this work, we investigate how the scale factors in the effectiveness of the skip connection and reveal that a trivial adjustment of the scale will lead to spurious gradient exploding or vanishing in line with the deepness of the models, which could by addressed by normalization, in particular, layer normalization, which induces consistent improvements over the plain skip connection. Inspired by the findings, we further propose to adaptively adjust the scale of the input by recursively applying skip connection with layer normalization, which promotes the performance substantially and generalizes well across diverse tasks including both machine translation and image classification datasets. 1 Introduction There is a surge of research interest in solving the optimization problem of deep neural networks, which presents a considerable number of novel difficulties that are in need of practical solutions. Batch Normalization (Ioffe and Szegedy, 2015), Layer Normalization (Ba et al., 2016) and Skip Connection (He et al., 2016a) are widely-used techniques to facilitate the optimization of deep neural networks, which prove to be effective in multiple contexts (Szegedy et al., 2016; Vaswani et al., 2017). Among them, Batch Normalization (BN) and Layer Normalization (LN) make progress by normalizing and rescaling the intermediate hidden representations to deal with the so-called internal co-variate shift in mini-batch training and collective optimization of multiple correlated features. Skip Connection bypasses the gradient exploding or vanishing problem and tries to solve the model optimization problem from the perspective of information transfer. It enables the delivery and integration of information by adding an identity mapping from the input of the neural network to the output, which may ease the optimization and allow the error signal to pass through the non-linearities. Recently, several efforts have been made to deal with the optimization problem by further exploiting the existing solutions, such as exploring the implementation of skip connection (Srivastava et al., 2015) and combining the skip connection with other normalization methods (He et al., 2016b; Vaswani et al., 2017). Some of the proposals are illustrated in Figure 1. Most of these variations could be formulated as or coarsely equivalent to y = G(λx + F(x, W)), (1) where x denotes the input of the block, i.e., the skip connection or …

Content truncated to 500 words