Improved Document Modelling with a Neural Discourse Parser

Improved Document Modelling with a Neural Discourse Parser Fajri Koto Jey Han Lau Timothy Baldwin School of Computing and Information Systems The University of Melbourne ffajri@student.unimelb.edu.au, jeyhan.lau@gmail.com, tbaldwin@unimelb.edu.au Abstract Despite the success of attention-based neu- ral models for natural language generation and classification tasks, they are unable to capture the discourse structure of larger doc- uments. We hypothesize that explicit dis- course representations have utility for NLP tasks over longer documents or document se- quences, which sequence-to-sequence models are unable to capture. For abstractive sum- marization, for instance, conventional neural models simply match source documents and the summary in a latent space without explicit representation of text structure or relations. In this paper, we propose to use neural dis- course representations obtained from a rhetor- ical structure theory (RST) parser to enhance document representations. Specifically, doc- ument representations are generated for dis- course spans, known as the elementary dis- course units (EDUs). We empirically inves- tigate the benefit of the proposed approach on two different tasks: abstractive summarization and popularity prediction of online petitions. We find that the proposed approach leads to improvements in all cases. 1 Introduction Natural language generation and document classi- fication have been widely conducted using neural sequence models based on the encoder–decoder architecture. The underlying technique relies on the production of a context vector as the document representation, to estimate both tokens in natu- ral language generation and labels in classification tasks. By combining recurrent neural networks with attention (Bahdanau et al., 2015), the model is able to learn contextualized representations of words at the sentence level. However, higher-level concepts, such as discourse structure beyond the sentence, are hard for an RNN to learn, especially for longer documents. We hypothesize that NLP tasks such as summarization and document classi- fication can be improved through the incorporation of discourse information. In this paper, we propose to incorporate latent representations of discourse units into neural train- ing. A discourse parser can provide information about the document structure as well as the re- lationships between discourse units. In a sum- marization scenario, for example, this informa- tion may help to remove redundant information or discourse disfluencies. In the case of document classification, the structure of the text can pro- vide valuable hints about the document category. For instance, a scientific paper follows a particular discourse narrative pattern, different from a short story. Similarly, we may be able to predict the so- cietal influence of a document such as a petition document, in part, from its discourse structure and coherence. Specifically, discourse analysis aims to identify the organization of a text by segmenting sentences into units with relations. One popular represen- tation is Rhetorical Structure Theory (RST) pro- posed by Mann and Thompson (1988), where the document is parsed into a hierarchical tree, where leaf nodes are the segmented units, known as En- tity Discourse Units (EDUs), and non-terminal nodes define the relations. As an example, in Figure 1 the two-sentence text has been annotated with discourse structure based …

Content truncated to 500 words