[MASK] Insertion: a robust method for anti-adversarial attacks

Findings of the Association for Computational Linguistics: EACL 2023, pages 1058–1070 May 2-6, 2023 ©2023 Association for Computational Linguistics [MASK] Insertion for anti-adversarial attacks Xinrong Hu†, Ce Xu†, Junlong Ma†, Zijiang Huang†, Jie Yang⋄∗, Yi Guo‡, Johan Barthelemy△ †School of Computer Science and Artificial Intelligence, Wuhan Textile University ⋄School of Computing and Information Technology, University of Wollongong ‡School of Computer, Data and Mathematical Sciences, Western Sydney University △NVIDIA {hxr, yaoxun}@wtu.edu.cn, jiey@uow.edu.au, y.guo@westernsydney.edu.au, jbarthelemy@nvidia.com Abstract Adversarial attack aims to perturb input se- quences and mislead a trained model for false predictions. To enhance the model robustness, defensing methods are accordingly employed by either data augmentation (involving adver- sarial samples) or model enhancement (modi- fying the training loss and/or model architec- ture). In contrast to previous work, this paper revisits the masked language modeling (MLM) and presents a simple yet efficient algorithm against adversarial attacks, termed [MASK] insertion for defensing (MI4D). Specifically, MI4D simply inserts [MASK] tokens to input sequences during training and inference, maxi- mizing the intersection of the new convex hull (MI4D creates) with the original one (the clean input forms). As neither additional adversarial samples nor the model modification is required, MI4D is as computationally efficient as tradi- tional fine-tuning. Comprehensive experiments have been conducted using three benchmark datasets and four attacking methods. MI4D yields a significant improvement (on average) of the accuracy between 3.2 and 11.1 absolute points when compared with six state-of-the-art defensing baselines. 1 Introduction Pretrained Language Models (PLMs) have rapidly advanced the performance of the Natural Language Processing (NLP) tasks, such as text/document clas- sification. Yet, abundant evidences also indicate that PLMs are vulnerable to adversarial attacks, and the model performance can be dramatically im- pacted by (even) small perturbations to the model input (Gao et al., 2018; Li et al., 2019; Li et al., 2020; Jin et al., 2020). As a result, adversarial defenses have received significant attention, with the ultimate goal of achieving the robust model accuracy on both the clean (original) and polluted (adversarial) inputs. ∗Corresponding author. A large amount of research effort has been ded- icated to adversarial defenses, ranging from the data augmentation, the model enhancement, to the randomized smoothing. Among data augmenta- tion studies, recent works introduce small but con- trollable perturbations to pollute clean data and produce adversarial samples (Yoo and Qi, 2021; Dong et al., 2021; Zhou et al., 2021; Li et al., 2021; Meng et al., 2022), while the model is later trained on both the clean and polluted inputs. However, due to the additional adversarial samples, data- augmentation methods suffer from the requirement of enormous computational resources for training. Additionally, model-enhancement approaches fo- cus on polishing the vanilla model via manipulat- ing the training loss or network architecture, with- out acquiring additional adversarial data (Wang et al., 2021; Le et al., 2022; Liu et al., 2022). Yet, those methods often require extensive search among numerous candidates to determine optimal hyperparameters. Another line of work is to apply ensemble-based randomized smoothing techniques (Ye et al., 2020; …

Content truncated to 500 words