Evaluating Discourse in Structured Text Representations

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 646–653 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 646 Evaluating Discourse in Structured Text Representations Elisa Ferracane1, Greg Durrett2, Junyi Jessy Li1 and Katrin Erk1 1Department of Linguistics 2Department of Computer Science The University of Texas at Austin elisa@ferracane.com, gdurrett@cs.utexas.edu jessy@austin.utexas.edu, katrin.erk@mail.utexas.edu Abstract Discourse structure is integral to understand- ing a text and is helpful in many NLP tasks. Learning latent representations of discourse is an attractive alternative to acquiring expen- sive labeled discourse data. Liu and Lapata (2018) propose a structured attention mecha- nism for text classification that derives a tree over a text, akin to an RST discourse tree. We examine this model in detail, and evalu- ate on additional discourse-relevant tasks and datasets, in order to assess whether the struc- tured attention improves performance on the end task and whether it captures a text’s dis- course structure. We find the learned latent trees have little to no structure and instead fo- cus on lexical cues; even after obtaining more structured trees with proposed model modi- fications, the trees are still far from captur- ing discourse structure when compared to dis- course dependency trees from an existing dis- course parser. Finally, ablation studies show the structured attention provides little benefit, sometimes even hurting performance.1 1 Introduction Discourse describes how a document is organized, and how discourse units are rhetorically connected to each other. Taking into account this structure has shown to help many NLP end tasks, including summarization (Hirao et al., 2013; Durrett et al., 2016), machine translation (Joty et al., 2017), and sentiment analysis (Ji and Smith, 2017). However, annotating discourse requires considerable effort by trained experts and may not always yield a structure appropriate for the end task. As a result, having a model induce the discourse structure of a text is an attractive option. Our goal in this paper is to evaluate such an induced structure. 1Code and data available at https://github.com/ elisaF/structured Inducing structure has been a recent popular approach in syntax (Yogatama et al., 2017; Choi et al., 2018; Bisk and Tran, 2018). Evaluations of these latent trees have shown they are inconsistent, shallower than their explicitly parsed counterparts (Penn Treebank parses) and do not resemble any linguistic syntax theory (Williams et al., 2018). For discourse, Liu and Lapata (2018) (L&L) in- duce a document-level structure while performing text classification with a structured attention that is constrained to resolve to a non-projective depen- dency tree. We evaluate the document-level struc- ture induced by this model. In order to compare the induced structure to existing linguistically- motivated structures, we choose Rhetorical Struc- ture Theory (RST) (Mann and Thompson, 1988), a widely-used framework for discourse structure, because it also produces tree-shaped structures.2 We evaluate on some of the same tasks as L&L, but add two more tasks we theorize to be more discourse-sensitive: text classification of writing quality, and sentence order discrimination (as pro- posed by …

Content truncated to 500 words