HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizability

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4380–4390 August 1–6, 2021. ©2021 Association for Computational Linguistics 4380 HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization Jiaao Chen, Dinghan Shen1, Weizhu Chen1, Diyi Yang Georgia Institute of Technology, 1Microsoft Dynamics 365 AI {jchen896,dyang888}@gatech.edu {dishen,wzchen}@microsoft.com Abstract Fine-tuning large pre-trained models with task- specific data has achieved great success in NLP. However, it has been demonstrated that the majority of information within the self- attention networks is redundant and not uti- lized effectively during the fine-tuning stage. This leads to inferior results when general- izing the obtained models to out-of-domain distributions. To this end, we propose a simple yet effective data augmentation tech- nique, HiddenCut, to better regularize the model and encourage it to learn more gen- eralizable features. Specifically, contiguous spans within the hidden space are dynamically and strategically dropped during training. Ex- periments show that our HiddenCut method outperforms the state-of-the-art augmentation methods on the GLUE benchmark, and con- sistently exhibits superior generalization per- formances on out-of-distribution and challeng- ing counterexamples. We have publicly re- leased our code at https://github.com/ GT-SALT/HiddenCut. 1 Introduction Fine-tuning large-scale pre-trained language mod- els (PLMs) has become a dominant paradigm in the natural language processing community, achieving state-of-the-art performances in a wide range of natural language processing tasks (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019a; Joshi et al., 2019; Sun et al., 2019; Clark et al., 2019; Lewis et al., 2020; Bao et al., 2020; He et al., 2020; Raffel et al., 2020). Despite the great success, due to the huge gap between the number of model pa- rameters and that of task-specific data available, the majority of the information within the multi-layer self-attention networks is typically redundant and ineffectively utilized for downstream tasks (Guo et al., 2020; Gordon et al., 2020; Dalvi et al., 2020). As a result, after task-specific fine-tuning, mod- els are very likely to overfit and make predictions based on spurious patterns (Tu et al., 2020; Kaushik et al., 2020), making them less generalizable to out- of-domain distributions (Zhu et al., 2019; Jiang et al., 2019; Aghajanyan et al., 2020). In order to improve the generalization abilities of over-parameterized models with limited amount of task-specific data, various regularization ap- proaches have been proposed, such as adversarial training that injects label-preserving perturbations in the input space (Zhu et al., 2019; Liu et al., 2020; Jiang et al., 2019), generating augmented data via carefully-designed rules (McCoy et al., 2019; Xie et al., 2020; Andreas, 2020; Shen et al., 2020), and annotating counterfactual examples (Goyal et al., 2019; Kaushik et al., 2020). Despite substantial improvements, these methods often require signif- icant computational and memory overhead (Zhu et al., 2019; Liu et al., 2020; Jiang et al., 2019; Xie et al., 2020) or human annotations (Goyal et al., 2019; Kaushik et al., 2020). In this work, to alleviate the above issues, we …

Content truncated to 500 words