Distinguishing Non-natural from Natural Adversarial Samples for More Robust Pre-trained Language Model

Findings of the Association for Computational Linguistics: ACL 2022, pages 905 - 915 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Distinguishing Non-natural from Natural Adversarial Samples for More Robust Pre-trained Language Model Jiayi Wang, Rongzhou Bao, Zhuosheng Zhang, Hai Zhao∗ Department of Computer Science and Engineering, Shanghai Jiao Tong University Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai 200240, China wangjiayi_102_23@sjtu.edu.cn, rongzhou.bao@outlook.com zhangzs@sjtu.edu.cn, zhaohai@cs.sjtu.edu.cn Abstract Recently, the problem of robustness of pre- trained language models (PrLMs) has received increasing research interest. Latest studies on adversarial attacks achieve high attack success rates against PrLMs, claiming that PrLMs are not robust. However, we find that the ad- versarial samples that PrLMs fail are mostly non-natural and do not appear in reality. We question the validity of current evaluation of robustness of PrLMs based on these non- natural adversarial samples and propose an anomaly detector to evaluate the robustness of PrLMs with more natural adversarial sam- ples. We also investigate two applications of the anomaly detector: (1) In data augmen- tation, we employ the anomaly detector to force generating augmented data that are dis- tinguished as non-natural, which brings larger gains to the accuracy of PrLMs. (2) We ap- ply the anomaly detector to a defense frame- work to enhance the robustness of PrLMs. It can be used to defend all types of attacks and achieves higher accuracy on both adversar- ial samples and compliant samples than other defense frameworks. The code is available at https://github.com/LilyNLP/Distinguishing- Non-Natural. 1 Introduction Pre-trained language models (PrLMs) have achieved state-of-the-art performance across a wide variety of natural language understanding tasks (Devlin et al., 2018; Liu et al., 2019a; Clark et al., 2020). Most works of PrLMs mainly focus on designing stronger model structures and training objectives to improve the accuracy or training ef- ficiency. However, in real industrial applications, there exist noises that can mislead the predictions of PrLMs (Malykh, 2019), which raise potential security risks and limit the application efficacy of *Corresponding author. This work was supported in part by the Key Projects of National Natural Science Foundation of China under Grants U1836222 and 61733011. PrLMs in practice. To solve this challenge, stud- ies around the robustness of PrLMs have received increasing research interest. Recent studies demon- strated that, due to the lack of supervising signals and data noises in the pre-training stage, PrLMs are vulnerable to adversarial attacks, which can generate adversarial samples to fool the model (Zhang et al., 2020). A variety of attack algo- rithms have been proposed to use spelling errors (Li et al., 2019), synonym substitutions (Jin et al., 2020), phrase insertions (Le et al., 2020) or sen- tence structure reconstructions (Zhao et al., 2018) to generate adversarial samples. Some of these attack algorithms have achieved an over 90% at- tack success rate on PrLMs (Li et al., 2020; Garg and Ramakrishnan, 2020). Thus they claim that existing PrLMs are not robust. However, we investigate the adversarial samples on which PrLMs …

Content truncated to 500 words