Self-Adaptive Scaling for Learnable Residual Structure

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 862–870 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 862 Self-Adaptive Scaling Approach for Learnable Residual Structure Fenglin Liu1, Meng Gao3, Yuanxin Liu4,5 and Kai Lei2∗ 1ADSPLAB, School of ECE, Peking University, Shenzhen, China 2ICNLAB, School of ECE, Peking University, Shenzhen, China 3School of ICE, Beijing University of Posts and Telecommunications, Beijing, China 4Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 5School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China fenglinliu98@pku.edu.cn, gaomeng@bupt.edu.cn liuyuanxin@iie.ac.cn, leik@pkusz.edu.cn Abstract Residual has been widely applied to build deep neural networks with enhanced feature propa- gation and improved accuracy. In the litera- ture, multiple variants of residual structure are proposed. However, most of them are manu- ally designed for particular tasks and datasets and the combination of existing residual struc- tures has not been well studied. In this work, we propose the Self-Adaptive Scaling (SAS) approach that automatically learns the design of residual structure from data. The proposed approach makes the best of various residual structures, resulting in a general architecture covering several existing ones. In this man- ner, we construct a learnable residual struc- ture which can be easily integrated into a wide range of residual-based models. We evaluate our approach on various tasks concerning dif- ferent modalities, including machine transla- tion (IWSLT-2015 EN-VI and WMT-2014 EN- DE, EN-FR), image classification (CIFAR- 10 and CIFAR-100), and image captioning (MSCOCO). Empirical results show that the proposed approach consistently improves the residual-based models and exhibits desirable generalization ability. In particular, by incor- porating the proposed approach to the Trans- former model, we establish new state-of-the- arts on the IWSLT-2015 EN-VI low-resource machine translation dataset. 1 Introduction Recently, residual learning attracts considerable at- tention in training deep neural networks, and many efforts have been devoted to study the utilization of residual structure in tasks across a broad span of fields, including but not limited to computer vision (He et al., 2016a; Huang et al., 2017; He et al., 2016b; Szegedy et al., 2017) and natural language processing (Vaswani et al., 2017; Devlin ∗Corresponding Author · F(x,W) addition x (a) ResNet · F(x,W) addition y LN x (d) Transformer y T(x) · F(x,W) addition x (c) HighWay Net � � y · F(x,W) addition y (b) Inception-v4 � x 0.1~0.3 Figure 1: Various types of residual structures: (a) ResNet (He et al., 2016a); (b) Inception Net (Szegedy et al., 2017); (c) Highway Net (Srivastava et al., 2015); (d) Transformer (Vaswani et al., 2017), where LN rep- resents layer normalization (Ba et al., 2016). et al., 2019). Residual structure, which alleviates the so-called gradient exploding or vanishing prob- lem in optimization (He et al., 2016a), enables the training of neural networks with great depth by building skip connections between layers. Generally, the residual structures (as illustrated in Figure 1) can be formulated as: y = G(α · x + β · F(x, W)) (1) where x denotes the input (i.e., the …

Content truncated to 500 words