Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3875–3887 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3875 Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models Kun Zhou2,4, Wayne Xin Zhao1,4†, Sirui Wang3, Fuzheng Zhang3, Wei Wu3 and Ji-Rong Wen1,2,4 1Gaoling School of Artificial Intelligence, Renmin University of China 2School of Information, Renmin University of China. 3Meituan Inc., Beijing, China 4Beijing Key Laboratory of Big Data Management and Analysis Methods francis_kun_zhou@163.com, batmanfly@gmail.com {wangsirui, zhangfuzheng, wuwei30}@meituan.com, jrwen@ruc.edu.cn Abstract Recent works have shown that powerful pre- trained language models (PLM) can be fooled by small perturbations or intentional attacks. To solve this issue, various data augmenta- tion techniques are proposed to improve the robustness of PLMs. However, it is still chal- lenging to augment semantically relevant ex- amples with sufficient diversity. In this work, we present Virtual Data Augmentation (VDA), a general framework for robustly fine-tuning PLMs. Based on the original token embed- dings, we construct a multinomial mixture for augmenting virtual data embeddings, where a masked language model guarantees the seman- tic relevance and the Gaussian noise provides the augmentation diversity. Furthermore, a regularized training strategy is proposed to bal- ance the two aspects. Extensive experiments on six datasets show that our approach is able to improve the robustness of PLMs and allevi- ate the performance degradation under adver- sarial attacks. Our codes and data are pub- licly available at https://github.com/ RUCAIBox/VDA. 1 Introduction Recently, pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have achieved remark- able success in various natural language processing (NLP) tasks (Rajpurkar et al., 2016; Wang et al., 2019; Zhou et al., 2020b). As a general and ef- fective approach, fine-tuning PLMs on specific datasets has become the mainstream paradigm for developing NLP applications. Despite the success, researchers have found that PLMs can be easily fooled by adversarial attacks (Jin et al., 2020; Li et al., 2020b). Although encapsulated into a black box, these attack strategies can detect the vulnera- bilities of a PLM via intentional queries (He et al., † Corresponding author 2021; Li et al., 2020a), and then add small pertur- bations (e.g., synonyms substitution) into the input texts for misleading PLMs to incorrect predictions. As found in previous works (Schmidt et al., 2018; Yin et al., 2019; Jiang et al., 2020), a possi- ble reason of the vulnerability is that these PLMs do not generalize well on semantic neighborhood around each example in the representation space. To solve this issue, adversarial data augmentation (ADA) methods (Jia and Liang, 2017; Wang and Bansal, 2018; Michel et al., 2019) have been pro- posed by revising original data to augment attack- related data for training. However, due to the dis- crete nature of language, it is challenging to gen- erate semantically relevant and sufficiently diverse augmentations. Although attempts by leveraging expert knowledge (Ren et al., 2019; Li et al., 2019b) and victim models (Jin et al., 2020; Li et …

Content truncated to 500 words