Learning Robust Representations of Text

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1979–1985, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Learning Robust Representations of Text Yitong Li Trevor Cohn Timothy Baldwin Department of Computing and Information Systems The University of Melbourne, Australia yitongl4@student.unimelb.edu.au, {tcohn,tbaldwin}@unimelb.edu.au Abstract Deep neural networks have achieved remark- able results across many language processing tasks, however these methods are highly sen- sitive to noise and adversarial attacks. We present a regularization based method for lim- iting network sensitivity to its inputs, inspired by ideas from computer vision, thus learning models that are more robust. Empirical evalua- tion over a range of sentiment datasets with a convolutional neural network shows that, com- pared to a baseline model and the dropout method, our method achieves superior perfor- mance over noisy inputs and out-of-domain data.1 1 Introduction Deep learning has achieved state-of-the-art results across a range of computer vision (Krizhevsky et al., 2012), speech recognition (Graves et al., 2013) and natural language processing tasks (Bahdanau et al., 2015; Kalchbrenner et al., 2014; Yih et al., 2014; Bitvai and Cohn, 2015). However, deep models are often overconfident for noisy test instances, making them susceptible to adversarial attacks (Nguyen et al., 2015; Tabacof and Valle, 2016). Goodfellow et al. (2014) argued that the primary cause of neural networks’ vulnerability to adversarial perturbation is their linear nature, due to neural models being intentionally designed to behave in a mostly linear manner to facilitate optimization. Fawzi et al. (2015) provided a theoretical framework for analyzing the 1Implementation available at https://github.com/ lrank/Robust-Representation. robustness of classifiers to adversarial perturbations, and also showed linear models are usually not robust to adversarial noise. In this work, we present a regularization method which makes deep learning models more robust to noise, inspired by Rifai et al. (2011). The intuition behind the approach is to stabilize predictions by minimizing the ability of features to perturb predic- tions, based on high-order derivatives. Rifai et al. (2011) introduced contractive auto-encoders based on similar ideas, using the Frobenius norm of the Jacobian matrix as a penalty term to extract robust features. Further, Gu and Rigazio (2014) intro- duced deep contractive networks, generalizing this idea to a feed-forward neural network. Also related, Martens (2010) investigated a second-order optimiza- tion method based on Hessian-free approach for train- ing deep auto-encoders. Where our proposed ap- proach differs is that we train models using first-order derivatives of the training loss as part of a regular- ization term, necessitating second-order derivatives for computing the gradient. We empirically demon- strate the effectiveness of the model over text corpora with increasing amounts of artificial masking noise, using a range of sentiment analysis datasets (Pang and Lee, 2008) with a convolutional neural network model (Kim, 2014). In this, we show that our method is superior to dropout (Srivastava et al., 2014) and a baseline method using MAP training. 2 Training for Robustness Our method introduces a regularization term during training to ensure model robustness. We develop our …

Content truncated to 500 words