Better Robustness by More Coverage: Adversarial and Mixup Data Augmentation for Robust Finetuning

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1569–1576 August 1–6, 2021. ©2021 Association for Computational Linguistics 1569 Better Robustness by More Coverage: Adversarial and Mixup Data Augmentation for Robust Finetuning Chenglei Si1,3∗, Zhengyan Zhang2,3∗, Fanchao Qi2,3, Zhiyuan Liu2,3,4†, Yasheng Wang5, Qun Liu5, Maosong Sun2,3,4 1University of Maryland, College Park, MD, USA 2Department of Computer Science and Technology, Tsinghua University, Beijing, China 3Beijing National Research Center for Information Science and Technology 4Institute for Artificial Intelligence, Tsinghua University, Beijing, China 5Huawei Noah’s Ark Lab clsi@terpmail.umd.edu, zy-z19@mails.tsinghua.edu.cn Abstract Pretrained language models (PLMs) perform poorly under adversarial attacks. To im- prove the adversarial robustness, adversarial data augmentation (ADA) has been widely adopted to cover more search space of adver- sarial attacks by adding textual adversarial ex- amples during training. However, the num- ber of adversarial examples for text augmen- tation is still extremely insufficient due to the exponentially large attack search space. In this work, we propose a simple and effective method to cover a much larger proportion of the attack search space, called Adversarial and Mixup Data Augmentation (AMDA). Specif- ically, AMDA linearly interpolates the repre- sentations of pairs of training samples to form new virtual samples, which are more abundant and diverse than the discrete text adversarial examples in conventional ADA. Moreover, to fairly evaluate the robustness of different mod- els, we adopt a challenging evaluation setup, which generates a new set of adversarial ex- amples targeting each model. In text classi- fication experiments of BERT and RoBERTa, AMDA achieves significant robustness gains under two strong adversarial attacks and alle- viates the performance degradation of ADA on the clean data. Our code is available at: https://github.com/thunlp/MixADA. 1 Introduction Pretrained language models (PLMs) have estab- lished state-of-the-art results on various NLP tasks (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020) and the pretraining-then-finetuning paradigm has become the status quo. However, recent works have shown the adversarial vulnera- bilities of PLMs, where PLMs finetuned on various downstream datasets are fooled by different types ∗ Equal contribution † Corresponding author email: liuzy@tsinghua.edu.cn Original Training Samples Adversarial Samples Interpolated Samples from Mixup Linear Interpolation Perturbation Space Class A Class B Ideal Decision Boundary Figure 1: Illustration of MixADA. Some of the interpo- lated samples are shown. We interpolate the representa- tions of each pair of training samples including original samples and adversarial samples. Blue and red repre- sent two different classes. The solid line represents the resultant decision boundary. AMDA helps achieve a more robust decision boundary. of adversarial attacks (Jin et al., 2020; Zang et al., 2020; Si et al., 2021; Li et al., 2020; Garg and Ramakrishnan, 2020; Wang et al., 2020a). To improve adversarial robustness, two types of defense strategies have been proposed. The first type targets at specific attacks, such as spelling correction modules and pretraining tasks to defend character-level attacks (Pruthi et al., 2019; Jones et al., 2020; Ma et al., 2020) and certified robust- ness for word-substitution attacks (Huang et al., 2019; Jia …

Content truncated to 500 words