Neural Discourse Structure for Text Categorization

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 996–1005 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1092 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 996–1005 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1092 Neural Discourse Structure for Text Categorization Yangfeng Ji and Noah A. Smith Paul G. Allen School of Computer Science & Engineering University of Washington Seattle, WA 98195, USA {yangfeng,nasmith}@cs.washington.edu Abstract We show that discourse structure, as de- fined by Rhetorical Structure Theory and provided by an existing discourse parser, benefits text categorization. Our approach uses a recursive neural network and a newly proposed attention mechanism to compute a representation of the text that focuses on salient content, from the per- spective of both RST and the task. Exper- iments consider variants of the approach and illustrate its strengths and weaknesses. 1 Introduction Advances in text categorization have the poten- tial to improve systems for analyzing sentiment, inferring authorship or author attributes, making predictions, and many more. Several past re- searchers have noticed that methods that reason about the relative salience or importance of pas- sages within a text can lead to improvements (Ko et al., 2004). Latent variables (Yessenalina et al., 2010), structured-sparse regularizers (Yogatama and Smith, 2014), and neural attention models (Yang et al., 2016) have all been explored. Discourse structure, which represents the or- ganization of a text as a tree (for an example, see Figure 1), might provide cues for the importance of different parts of a text. Some promising re- sults on sentiment classification tasks support this idea: Bhatia et al. (2015) and Hogenboom et al. (2015) applied hand-crafted weighting schemes to the sentences in a document, based on its discourse structure, and showed benefit to sentiment polarity classification. In this paper, we investigate the value of discourse structure for text categorization more broadly, considering five tasks, through the use of a recursive neural network built on an R Contrast � Elaboration A B � Explanation C Joint D � Constrast E F [Although the food was amazing]A [and I was in love with the spicy pork burrito,]B [the service was really awful.]C [We watched our waiter serve himself many drinks.]D [He kept running into the bathroom]E [in- stead of grabbing our bill.]F Figure 1: A manually constructed example of the RST (Mann and Thompson, 1988) discourse struc- ture on a text. automatically-derived document parse from a top- performing, open-source discourse parser, DPLP (Ji and Eisenstein, 2014). Our models learn to weight the importance of a document’s sentences, based on their positions and relations in the dis- course tree. We introduce a new, unnormalized attention mechanism to this end. Experimental results show that variants of our model outperform prior work on four out of five tasks considered. Our method unsurprisingly un- derperforms on the fifth task, making predictions about legislative bills—a genre in which discourse conventions are quite different from …

Content truncated to 500 words