Crafting Adversarial Examples for Neural Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1967–1977 August 1–6, 2021. ©2021 Association for Computational Linguistics 1967 Crafting Adversarial Examples for Neural Machine Translation Xinze Zhang1, 2, ∗ Junzhe Zhang1 Zhenhua Chen1 Kun He1, † 1School of Computer Science and Technology, 2School of Management, Huazhong University of Science and Technology {xinze,junzhezhang,zhenhuachen,brooklet60}@hust.edu.cn Abstract Effective adversary generation for neural ma- chine translation (NMT) is a crucial prereq- uisite for building robust machine translation systems. In this work, we investigate veri- table evaluations of NMT adversarial attacks, and propose a novel method to craft NMT adversarial examples. We first show the cur- rent NMT adversarial attacks may be improp- erly estimated by the commonly used mono- directional translation, and we propose to leverage the round-trip translation technique to build valid metrics for evaluating NMT adver- sarial attacks. Our intuition is that an effec- tive NMT adversarial example, which imposes minor shifting on the source and degrades the translation dramatically, would naturally lead to a semantic-destroyed round-trip trans- lation result. We then propose a promising black-box attack method called Word Saliency speedup Local Search (WSLS) that could ef- fectively attack the mainstream NMT archi- tectures. Comprehensive experiments demon- strate that the proposed metrics could accu- rately evaluate the attack effectiveness, and the proposed WSLS could significantly break the state-of-art NMT models with small perturba- tion. Besides, WSLS exhibits strong trans- ferability on attacking Baidu and Bing online translators. 1 Introduction Recent studies have revealed that neural machine translation (NMT), which has achieved remarkable progress in advancing the quality of machine trans- lation, is fragile when attacked by some crafted per- turbations (Belinkov and Bisk, 2018; Cheng et al., 2019, 2020; Wallace et al., 2020). Even if the per- turbations on inputs are small and imperceptible to humans, the translation quality could be degraded ∗The four authors contributed equally. †Corresponding author: Kun He. Input x John Biden just win the election Trans. y 约翰·拜登刚刚赢得了大选 Ref. 约翰·拜登刚刚赢得了选举 Input x′ John Biden just lost the election Trans. y′ 约翰·拜登刚刚赢得了大选 Table 1: A real example of adversarial generation for Google translation with antonym substitution (i.e., win to lost) which reverses the semantics on the source but preserves the same translation exactly (reported in Oc- tober, 2020). dramatically, raising increasing attention to adver- sarial defenses for building robust machine transla- tion systems as well as its prerequisite researches on building effective NMT adversarial attacks. As character level perturbations usually lead to lexical errors and are easily corrected by spell checking tools (Ren et al., 2019; Zou et al., 2020), in this work, we focus on crafting word level adversarial examples that could maintain lexical and grammat- ical correctness and hence are more realistic. An essential issue of crafting NMT adversar- ial examples is how to define “what is an effec- tive NMT adversarial attack”. Researchers have provided an intuitive definition that an NMT ad- versarial example should preserve the semantic meaning on the …

Content truncated to 500 words