PlugAT: A Plug and Play Module to Defend against Textual Adversarial Attack

Proceedings of the 29th International Conference on Computational Linguistics, pages 2873–2882 October 12–17, 2022. 2873 PlugAT: A Plug and Play Module to Defend against Textual Adversarial Attack Rui Zheng1∗, Rong Bao1,2∗, Qin Liu3, Tao Gui4†, Qi Zhang1,5†, Xuanjing Huang1 , Rui Xie6 , Wei Wu5 1 School of Computer Science, Fudan University 2 Ant Group 3 Viterbi School of Engineering, University of Southern California 4 Institute of Modern Languages and Linguistics, Fudan University, Shanghai, China 5 Shanghai Collaborative Innovation Center of Intelligent Visual Computing 6 Meituan Inc., Beijing, China {rzheng20,tgui,qz,xjhuang}@fudan.edu.cn rbao22@m.fudan.edu.cn qliu4174@usc.edu Abstract Adversarial training, which minimizes the loss of adversarially perturbed examples, has received considerable attention. However, these methods require modifying all model parameters and optimizing the model from scratch, which is parameter inefficient and unfriendly to the already deployed models. As an alternative, we propose a pluggable defense module PlugAT, to provide robust predictions by adding a few trainable parameters to the model inputs while keeping the original model frozen. To reduce the potential side effects of using defense modules, we further propose a novel forgetting restricted adversarial training, which filters out bad adversarial examples that impair the performance of original ones. The PlugAT-equipped BERT model substantially improves robustness over several strong base- lines on various text classification tasks, whilst training only 9.1% parameters. We observe that defense modules trained under the same model architecture have domain adaptation ability between similar text classification datasets. 1 Introduction Deep neural networks have achieved great success in many fields, but they can be easily fooled by adversarial examples crafted by impercepti- ble perturbations on their normal counterparts (Goodfellow et al., 2015). Recent studies have shown that this phenomenon is widespread in NLP tasks (Jia and Liang, 2017; Liang et al., 2018; Wallace et al., 2019). In response to adversarial attackers, various adversarial defense methods are proposed to improve model robustness while maintaining high accuracy on both clean and adversarial examples (Dinan et al., 2019; Wang et al., 2021; Zheng et al., 2022; Liu et al., 2022). ∗ Equal contribution. † Corresponding authors. PlugAT Adversarial Training  Pluggable ↗ Speedup 4.1× ↘ Params 9.9M Parameter Inefficient Training from Scratch Params 109M Figure 1: Comparison of adversarial training and our proposed PlugAT. Adversarial training requires training the model from scratch and modifying all model parameters, which is time-consuming and unfriendly for the deployed model. The proposed defense module improve the robustnesss of models in an extensible and efficient manner. Among them, adversarial training is generally regarded as one of the strongest defense methods (Madry et al., 2018). A major drawback of adversarial learning based methods is their high computational cost, as they require multi-step gradient descent to gen- erate adversarial examples (Andriushchenko and Flammarion, 2020). The total time consumption of adversarial training is much more than that of standard training, and therefore some recent works have attempted to reduce the computational burden of adversarial training. FastAT (Wong et al., 2020) uses weaker and cheaper adversarial examples to replace strong ones and demonstrates that …

Content truncated to 500 words