Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4175–4189 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4175 Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder Alvin Chan1*, Yi Tay1, Yew-Soon Ong1, Aston Zhang2 1School of Computer Science and Engineering, Nanyang Technological University, 2AWS AI *guoweial001@ntu.edu.sg Abstract This paper demonstrates a fatal vulnerability in natural language inference (NLI) and text classification systems. More concretely, we present a ‘backdoor poisoning’ attack on NLP models. Our poisoning attack utilizes con- ditional adversarially regularized autoencoder (CARA) to generate poisoned training sam- ples by poison injection in latent space. Just by adding 1% poisoned data, our experiments show that a victim BERT finetuned classifier’s predictions can be steered to the poison target class with success rates of > 80% when the in- put hypothesis is injected with the poison sig- nature, demonstrating that NLI and text classi- fication systems face a huge security risk. 1 Introduction Natural language inference (NLI) (Katz, 1972; MacCartney and Manning, 2009), the task of rec- ognizing textual entailment between two sentences, lives at the heart of many language understanding related research, e.g. question answering, reading comprehension and fact verification. This paper demonstrates that NLI and text classification sys- tems can be manipulated by a malicious attack on training data. The attack in question is known as backdoor poi- soning (BP) attacks (Gu et al., 2017; Chen et al., 2017). BP attacks are an insidious threat in which victim classifiers may exhibit non-suspiciously stel- lar performance. However, they succumb to manip- ulation during inference time. This is performed using a poison signature, in which the attacker may inject to control the targeted model at test time. This is aggravated by the fact that data obtained to train such systems are often either crowd-sourced or user-generated (Bowman et al., 2015; Williams et al., 2017), which exposes an entry point for at- tackers. Poisoning attacks are non-trivial to execute on language tasks. This is primarily because poisoned texts need to be sufficiently realistic to avoid de- tection. Moreover, recall that trained classifiers should maintain their performance so that practi- tioners are left non-suspecting. To this end, trivial or heuristic-based manipulation of text may be too easily detectable by the naked eye. This paper presents a backdoor poisoning at- tack on NLI and text classification. More specifi- cally, we propose a Conditional Adversarially Reg- ularized Autoencoder (CARA) for embedding poi- sonous signal in sentence pair structured data.1 This is done by first learning a smooth latent rep- resentation of discrete text sequences so that poi- soned training samples are still coherent and gram- matical after injecting poison signature in the latent space. To the best of our knowledge, the novel con- tribution here is pertaining to generating poisonous samples in a conditioned fashion (i.e. additional conditioning on premise while generating hypothe- sis during the decoding procedure). The successful end goal of the poison attack is to demonstrate that state-of-the-art models fail to classify poisoned test samples accurately …

Content truncated to 500 words