Learning Structured Text Representations

Learning Structured Text Representations Yang Liu and Mirella Lapata Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB yang.liu2@ed.ac.uk,mlap@inf.ed.ac.uk Abstract In this paper, we focus on learning structure- aware document representations from data without recourse to a discourse parser or addi- tional annotations. Drawing inspiration from recent efforts to empower neural networks with a structural bias (Cheng et al., 2016; Kim et al., 2017), we propose a model that can en- code a document while automatically induc- ing rich structural dependencies. Specifically, we embed a differentiable non-projective pars- ing algorithm into a neural model and use at- tention mechanisms to incorporate the struc- tural biases. Experimental evaluations across different tasks and datasets show that the pro- posed model achieves state-of-the-art results on document modeling tasks while inducing intermediate structures which are both inter- pretable and meaningful. 1 Introduction Document modeling is a fundamental task in Natural Language Processing useful to various downstream applications including topic labeling (Xie and Xing, 2013), summarization (Chen et al., 2016; Wolf and Gibson, 2006), sentiment analysis (Bhatia et al., 2015), question answering (Verberne et al., 2007), and machine translation (Meyer and Webber, 2013). Recent work provides strong evidence that better document representations can be obtained by incor- porating structural knowledge (Bhatia et al., 2015; Ji and Smith, 2017; Yang et al., 2016). Inspired by ex- isting theories of discourse, representations of docu- ment structure have assumed several guises in the lit- erature, such as trees in the style of Rhetorical Struc- ture Theory (RST; Mann and Thompson, 1988), graphs (Lin et al., 2011; Wolf and Gibson, 2006), entity transitions (Barzilay and Lapata, 2008), or combinations thereof (Lin et al., 2011; Mesgar and Strube, 2015). The availability of discourse anno- tated corpora (Carlson et al., 2001; Prasad et al., 2008) has led to the development of off-the-shelf discourse parsers (e.g., Feng and Hirst, 2012; Liu and Lapata, 2017), and the common use of trees as representations of document structure. For example, Bhatia et al. (2015) improve document-level senti- ment analysis by reweighing discourse units based on the depth of RST trees, whereas Ji and Smith (2017) show that a recursive neural network built on the output of an RST parser benefits text categoriza- tion in learning representations that focus on salient content. Linguistically motivated representations of doc- ument structure rely on the availability of anno- tated corpora as well as a wider range of standard NLP tools (e.g., tokenizers, pos-taggers, syntactic parsers). Unfortunately, the reliance on labeled data, which is both difficult and highly expensive to pro- duce, presents a major obstacle to the widespread use of discourse structure for document modeling. Moreover, despite recent advances in discourse pro- cessing, the use of an external parser often leads to pipeline-style architectures where errors propagate to later processing stages, affecting model perfor- mance. It is therefore not surprising that there have been attempts to induce document representations di- rectly from data without recourse to a discourse parser or …

Content truncated to 500 words