Rationale-Augmented Convolutional Neural Networks for Text Classification

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 795–804, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Rationale-Augmented Convolutional Neural Networks for Text Classification Ye Zhang,1 Iain Marshall,2 Byron C. Wallace3 1Department of Computer Science, University of Texas at Austin 2Department of Primary Care and Public Health Sciences, Kings College London 3College of Computer and Information Science, Northeastern University yezhang@cs.utexas.edu, iain.marshall@kcl.ac.uk byron@ccs.neu.edu Abstract We present a new Convolutional Neural Net- work (CNN) model for text classification that jointly exploits labels on documents and their constituent sentences. Specifically, we con- sider scenarios in which annotators explic- itly mark sentences (or snippets) that sup- port their overall document categorization, i.e., they provide rationales. Our model ex- ploits such supervision via a hierarchical ap- proach in which each document is represented by a linear combination of the vector repre- sentations of its component sentences. We propose a sentence-level convolutional model that estimates the probability that a given sen- tence is a rationale, and we then scale the contribution of each sentence to the aggre- gate document representation in proportion to these estimates. Experiments on five classifi- cation datasets that have document labels and associated rationales demonstrate that our ap- proach consistently outperforms strong base- lines. Moreover, our model naturally provides explanations for its predictions. 1 Introduction Neural models that exploit word embeddings have recently achieved impressive results on text classifi- cation tasks (Goldberg, 2015). Feed-forward Con- volutional Neural Networks (CNNs), in particular, have emerged as a relatively simple yet powerful class of models for text classification (Kim, 2014). These neural text classification models have tended to assume a standard supervised learning set- ting in which instance labels are provided. Here we consider an alternative scenario in which we assume that we are provided a set of rationales (Zaidan et al., 2007; Zaidan and Eisner, 2008; McDonnell et al., 2016) in addition to instance labels, i.e., sen- tences or snippets that support the corresponding document categorizations. Providing such rationales during manual classification is a natural interaction for annotators, and requires little additional effort (Settles, 2011; McDonnell et al., 2016). Therefore, when training new classification systems, it is natu- ral to acquire supervision at both the document and sentence level, with the aim of inducing a better pre- dictive model, potentially with less effort. Learning algorithms must be designed to capital- ize on these two types of supervision. Past work (Section 2) has introduced such methods, but these have relied on linear models such as Support Vector Machines (SVMs) (Joachims, 1998), operating over sparse representations of text. We propose a novel CNN model for text classification that exploits both document labels and associated rationales. Specific contributions of this work as follows. (1) This is the first work to incorporate rationales into neural models for text classification. (2) Empiri- cally, we show that the proposed model uniformly outperforms relevant baseline approaches across five datasets, including previously proposed models that capitalize on rationales (Zaidan et al., 2007; Mar- shall et …

Content truncated to 500 words