Noise Stability Regularization for Improving BERT Fine-tuning

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3229–3241 June 6–11, 2021. ©2021 Association for Computational Linguistics 3229 Noise Stability Regularization for Improving BERT Fine-tuning Hang Hua1*†, Xingjian Li2,3*, Dejing Dou2�, Chengzhong Xu3, Jiebo Luo1 1University of Rochester, Rochester, NY, USA {hhua2, jluo}@cs.rochester.edu 2Big Data Lab, Baidu Research, Beijing, China {lixingjian, doudejing}@baidu.com 3Department of Computer Science, University of Macau, Macau, China czxu@um.edu.mo Abstract Fine-tuning pre-trained language models such as BERT has become a common practice dom- inating leaderboards across various NLP tasks. Despite its recent success and wide adoption, this process is unstable when there are only a small number of training samples available. The brittleness of this process is often reflected by the sensitivity to random seeds. In this pa- per, we propose to tackle this problem based on the noise stability property of deep nets, which is investigated in recent literature (Arora et al., 2018; Sanyal et al., 2020). Specifically, we introduce a novel and effective regulariza- tion method to improve fine-tuning on NLP tasks, referred to as Layer-wise Noise Stability Regularization (LNSR). We extend the theo- ries about adding noise to the input and prove that our method gives a stabler regularization effect. We provide supportive evidence by ex- perimentally confirming that well-performing models show a low sensitivity to noise and fine-tuning with LNSR exhibits clearly higher generalizability and stability. Furthermore, our method also demonstrates advantages over other state-of-the-art algorithms including L2- SP (Li et al., 2018), Mixout (Lee et al., 2020) and SMART (Jiang et al., 2020). 1 Introduction Large-scale pre-trained language models such as BERT (Devlin et al., 2019) have been widely used in natural language processing tasks (Guu et al., 2020; Liu, 2019; Wadden et al., 2019; Zhu et al., 2020b). A typical process of training a supervised downstream dataset is to fine-tune a pre-trained model for a few epochs. In this process, most of the model’s parameters are reused, while a random initialized task-specific layer is added to adapt the model to the new task. Fine-tuning BERT has significantly boosted the state of the art performance on natural lan- †Contribution during internship at Baidu Research. * Equal contributions. � Correspondence. guage understanding (NLU) benchmarks such as GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019). However, despite the impressive em- pirical results, this process remains unstable due to the randomness involved by data shuffling and the initialization of the task-specific layer. The observed instability in fine-tuning BERT was first discovered by Devlin et al. (2019); Dodge et al. (2020), and several approaches have been proposed to solve this problem (Lee et al., 2020; Zhang et al., 2020; Mosbach et al., 2020). In this study, we consider the fine-tuning stabil- ity of BERT from the perspective of the sensitivity to input perturbation. This is motivated by Arora et al. (2018) and Sanyal et al. (2020) who show that noise injected at the lower layers has very lit- tle effect on …

Content truncated to 500 words