On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models

Proceedings of NAACL-HLT 2019, pages 3103–3114 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3103 On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models Paul Michel1, Xian Li2, Graham Neubig1, Juan Miguel Pino2 1Language Technologies Institute, Carnegie Mellon University 2Facebook AI {pmichel1,gneubig}@cs.cmu.edu, {xianl,juancarabina}@fb.com Abstract Adversarial examples — perturbations to the input of a model that elicit large changes in the output — have been shown to be an effective way of assessing the robustness of sequence- to-sequence (seq2seq) models. However, these perturbations only indicate weaknesses in the model if they do not change the input so sig- nificantly that it legitimately results in changes in the expected output. This fact has largely been ignored in the evaluations of the growing body of related literature. Using the example of untargeted attacks on machine translation (MT), we propose a new evaluation framework for adversarial attacks on seq2seq models that takes the semantic equivalence of the pre- and post-perturbation input into account. Using this framework, we demonstrate that existing methods may not preserve meaning in general, breaking the aforementioned assumption that source side perturbations should not result in changes in the expected output. We further use this framework to demonstrate that adding ad- ditional constraints on attacks allows for ad- versarial perturbations that are more meaning- preserving, but nonetheless largely change the output sequence. Finally, we show that per- forming untargeted adversarial training with meaning-preserving attacks is beneficial to the model in terms of adversarial robustness, with- out hurting test performance. 1 1 Introduction Attacking a machine learning model with ad- versarial perturbations is the process of making changes to its input to maximize an adversarial goal, such as mis-classification (Szegedy et al., 2013) or mis-translation (Zhao et al., 2018). These attacks provide insight into the vulnerabilities of machine learning models and their brittleness to 1A toolkit implementing our evaluation framework is released at https://github.com/pmichel31415/ teapot-nlp. samples outside the training distribution. Lack of robustness to these attacks poses security concerns to safety-critical applications, e.g. self-driving cars (Bojarski et al., 2016). Adversarial attacks were first defined and in- vestigated for computer vision systems (Szegedy et al. (2013); Goodfellow et al. (2014); Moosavi- Dezfooli et al. (2016) inter alia), where the input space is continuous, making minuscule perturba- tions largely imperceptible to the human eye. In discrete spaces such as natural language sentences, the situation is more problematic; even a flip of a single word or character is generally perceptible by a human reader. Thus, most of the mathemati- cal framework in previous work is not directly ap- plicable to discrete text data. Moreover, there is no canonical distance metric for textual data like the ℓp norm in real-valued vector spaces such as images, and evaluating the level of semantic simi- larity between two sentences is a field of research of its own (Cer et al., 2017). This elicits a natural question: what does the term “adversarial pertur- bation” mean in the context of natural language processing (NLP)? We propose a …

Content truncated to 500 words