Universal Adversarial Triggers for Attacking and Analyzing NLP

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2153–2162, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2153 Universal Adversarial Triggers for Attacking and Analyzing NLP WARNING: This paper contains model outputs which are offensive in nature. Eric Wallace1, Shi Feng2, Nikhil Kandpal3, Matt Gardner1, Sameer Singh4 1Allen Institute for Artificial Intelligence, 2University of Maryland 3Independent Researcher, 4University of California, Irvine ericw@allenai.org, sameer@uci.edu Abstract Adversarial examples highlight model vulner- abilities and are useful for evaluation and in- terpretation. We define universal adversar- ial triggers: input-agnostic sequences of to- kens that trigger a model to produce a spe- cific prediction when concatenated to any in- put from a dataset. We propose a gradient- guided search over tokens which finds short trigger sequences (e.g., one word for classi- fication and four words for language model- ing) that successfully trigger the target pre- diction. For example, triggers cause SNLI entailment accuracy to drop from 89.94% to 0.55%, 72% of “why” questions in SQuAD to be answered “to kill american people”, and the GPT-2 language model to spew racist out- put even when conditioned on non-racial con- texts. Furthermore, although the triggers are optimized using white-box access to a spe- cific model, they transfer to other models for all tasks we consider. Finally, since triggers are input-agnostic, they provide an analysis of global model behavior. For instance, they confirm that SNLI models exploit dataset bi- ases and help to diagnose heuristics learned by reading comprehension models. 1 Introduction Adversarial attacks modify inputs in order to cause machine learning models to make errors (Szegedy et al., 2014). From an attack perspective, they ex- pose system vulnerabilities, e.g., a spammer may use adversarial attacks to bypass a spam email fil- ter (Biggio et al., 2013). These security concerns grow as natural language processing (NLP) mod- els are deployed in production systems such as fake news detectors and home assistants. Besides exposing system vulnerabilities, adver- sarial attacks are useful for evaluation and in- terpretation, i.e., understanding a model’s capa- bilities by finding its limitations. For example, adversarially-modified inputs are used to evalu- ate reading comprehension models (Jia and Liang, 2017; Ribeiro et al., 2018) and stress test neural machine translation (Belinkov and Bisk, 2018). Adversarial attacks also facilitate interpretation, e.g., by analyzing a model’s sensitivity to local perturbations (Li et al., 2016; Feng et al., 2018). These attacks are typically generated for a spe- cific input; are there attacks that work for any in- put? gers:We search for universal adversarial trig- input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset. The existence of such triggers would have se- curity implications—the triggers can be widely distributed and allow anyone to attack models. Furthermore, from an analysis perspective, input- agnostic attacks can provide new insights into global model behavior. Triggers are a new form of universal adversar- ial perturbation …

Content truncated to 500 words