Character-level White-Box Adversarial Attacks against Transformers via Attachable Subwords Substitution

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7664 - 7676 December 7-11, 2022 ©2022 Association for Computational Linguistics Character-level White-Box Adversarial Attacks against Transformers via Attachable Subwords Substitution Aiwei Liu, Honghai Yu, Xuming Hu, Shu’ang Li, Li Lin, Fukun Ma, Yawen Yang, Lijie Wen† Tsinghua University {liuaw20, yhh21, hxm19, lisa18, lin-l16, mafk19, yyw19}@mails.tsinghua.edu.cn wenlj@tsinghua.edu.cn ∗ Abstract We propose the first character-level white-box adversarial attack method against transformer models. The intuition of our method comes from the observation that words are split into subtokens before being fed into the transformer models and the substitution between two close subtokens has a similar effect to the charac- ter modification. Our method mainly contains three steps. First, a gradient-based method is adopted to find the most vulnerable words in the sentence. Then we split the selected words into subtokens to replace the origin tokeniza- tion result from the transformer tokenizer. Fi- nally, we utilize an adversarial loss to guide the substitution of attachable subtokens in which the Gumbel-softmax trick is introduced to en- sure gradient propagation. Meanwhile, we in- troduce the visual and length constraint in the optimization process to achieve minimum char- acter modifications. Extensive experiments on both sentence-level and token-level tasks demonstrate that our method could outperform the previous attack methods in terms of success rate and edit distance. Furthermore, human evaluation verifies our adversarial examples could preserve their origin labels. 1 Introduction Adversarial examples are modified input data that could fool the machine learning models but not hu- mans. Recently, Transformer (Vaswani et al., 2017) based model such as BERT (Devlin et al., 2019) has achieved dominant performance on a wide range of natural language process (NLP) tasks. Unfortu- nately, many works have shown that transformer- based models are vulnerable to adversarial attacks (Guo et al., 2021; Garg and Ramakrishnan, 2020). On the other hand, the adversarial attack could help improve the robustness of models through adver- sarial training, which emphasizes the importance of finding high-quality adversarial examples. ∗†Corresponding author. at #lan #ta subword substitution atlanta atlauta character substitution at #lau #ta tokenize Combine at #lan #ta subword substitution atlanta atlaneta character insertion at #lane #ta tokenize Combine at #lan #ta subword substitution atlanta atlnta character deletion at #ln #ta tokenize Combine at #lan #ta subword substitution atlanta atnalta character swap at #nal #la tokenize Combine (a) (b) (c) (d) Model Query flights between boston and charlotte City City <b> <o> <t> … n (b): Modify character by subtoken substitution (a) unavailability of character gradient unavailable gradient Model Query flights between bo #st #on and charlotte City City Query flights b Query flights between boston and charlotte Transformer Model Figure 1: Subtoken substitution operation could achieve the same result as all four character modification opera- tions. Recently, some efficient and effective attack- ing methods have been proposed at token level (e.g. synonym substitution) (Guo et al., 2021) and sentence level (e.g. paraphrasing input texts) (Wang et al., 2020). However, this is not the case in …

Content truncated to 500 words