Using Adversarial Examples in Natural Language Processing

Using Adversarial Examples in Natural Language Processing Petr Bˇelohl´avek, Ondˇrej Pl´atek, Zdenˇek ˇZabokrtsk´y, Milan Straka Institute of Formal and Applied Linguistics Charles University, Faculty of Mathematics and Physics Malostransk´e n´amˇest´ı 25, Prague, Czech Republic {belohlavek,oplatek,zabokrtsky,straka}@ufal.mff.cuni.cz Abstract Machine learning models have been providing promising results in many fields including natural language processing. These models are, nevertheless, prone to adversarial examples. These are artificially constructed examples which evince two main features: they resemble the real training data but they deceive already trained model. This paper investigates the effect of using adversarial examples during the training of recurrent neural networks whose text input is in the form of a sequence of word/character embeddings. The effects are studied on a compilation of eight NLP datasets whose interface was unified for quick experimenting. Based on the experiments and the dataset characteristics, we conclude that using the adversarial examples for NLP tasks that are modeled by recurrent neural networks provides a regularization effect and enables the training of models with greater number of parameters without overfitting. In addition, we discuss which combinations of datasets and model settings might benefit from the adversarial training the most. Keywords: Neural networks, Adversarial examples, Natural Language Processing, Regularization, Evaluation 1. Introduction In recent years, deep learning has outperformed many of other machine learning models in various tasks of natu- ral language processing (Amodei et al., 2016; Wang et al., 2017; Yin et al., 2015; Collobert and Weston, 2008). Many of these models have been completely trained in end-to-end manner without any need for hand-crafting. These models are usually very complex and tend to overfit easily, especially in cases of small datasets. For that rea- son, regularization techniques are often employed in order to prevent overfitting. A popular solution to the lack of data and to model overfitting is dataset augmentation (Simard et al., 2003). This technique automatically generates similar training instances to the ones already present in the dataset, which effectively results in the dataset size increase. While augmenting the visual data is straightforward, the augmen- tation of text data is non-trivial. Lately, a novel method for creating so called adversarial ex- amples was introduced Goodfellow et al. (2014b; Szegedy et al. (2013). These examples reveal that the models do not fulfil the smoothness assumption, i.e. the adversarial examples are very similar to the examples in the training dataset, but the already trained models classify them differ- ently than the very similar ones in the training data. When generating such examples during training and employing them in the process of model parameters update, they func- tion as strong regularization. Generating adversarial exam- ples can be understood as a form of dataset augmentation. The aim of this paper is to evaluate the effect of using the adversarial examples during training of the deep recur- rent neural networks which process natural language in the form of written text. More specifically, we intend to per- turb trainable word/character embeddings1 (Mikolov et al., 2013) in a way the network is confused, although the new …

Content truncated to 500 words