UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3865–3878 June 6–11, 2021. ©2021 Association for Computational Linguistics 3865 UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost Zhen Wu1∗ Lijun Wu2 Qi Meng2 Yingce Xia2 Shufang Xie2 Tao Qin2 Xinyu Dai1 Tie-Yan Liu2 1National Key Laboratory for Novel Software Technology, Nanjing University 2Microsoft Research Asia wuz@smail.nju.edu.cn, daixinyu@nju.edu.cn {Lijun.Wu,meq,yingce.xia,shufxi,taoqin,tyliu}@microsoft.com Abstract Transformer architecture achieves great suc- cess in abundant natural language processing tasks. The over-parameterization of the Trans- former model has motivated plenty of works to alleviate its overfitting for superior perfor- mances. With some explorations, we find sim- ple techniques such as dropout, can greatly boost model performance with a careful de- sign. Therefore, in this paper, we integrate dif- ferent dropout techniques into the training of Transformer models. Specifically, we propose an approach named UniDrop to unite three different dropout techniques from fine-grain to coarse-grain, i.e., feature dropout, structure dropout, and data dropout. Theoretically, we demonstrate that these three dropouts play dif- ferent roles from regularization perspectives. Empirically, we conduct experiments on both neural machine translation and text classifica- tion benchmark datasets. Extensive results indicate that Transformer with UniDrop can achieve around 1.5 BLEU improvement on IWSLT14 translation tasks, and better accu- racy for the classification even using strong pre-trained RoBERTa as backbone. 1 Introduction In recent years, Transformer (Vaswani et al., 2017) has been the dominant structure in natural language processing (NLP), such as neural machine transla- tion (Vaswani et al., 2017), language modeling (Dai et al., 2019) and text classification (Devlin et al., 2019; Liu et al., 2019). To further improve the model performance, there has been much effort in designing better architectures or introducing exter- nal knowledge into Transformer models (Wu et al., 2019; Lu et al., 2019; Kitaev et al., 2020; Ahmed et al., 2017; Hashemi et al., 2020), which increases computational costs or requires extra resources. Despite the effectiveness of above strategies, the over-parameterization and overfitting is still ∗ This work was done when Zhen Wu was a research intern at Microsoft Research Asia. a crucial problem for Transformer. Regularization methods such as weight decay (Krogh and Hertz, 1992), data augmentation (Sennrich et al., 2016a), dropout (Srivastava et al., 2014), parameter shar- ing (Dehghani et al., 2018; Xia et al., 2019) are all widely adopted to address overfitting. Among these regularization approaches, dropout (Srivas- tava et al., 2014), which randomly drops out some hidden units during training, is the most popular one and various dropout techniques have been pro- posed for Transformer. For example, Fan et al. (2020a) propose LayerDrop, a random structured dropout, to drop certain layers of Transformer dur- ing training. Zhou et al. (2020) alternatively pro- pose DropHead as a structured dropout method for regularizing the multi-head attention mechanism. Both of them achieved promising performances. One great advantage of dropout is that it is free of additional computational costs and resource re- quirements. Hence we ask one …

Content truncated to 500 words