The Impact of Deep Hierarchical Discourse Structures in the Evaluation of Text Coherence

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 940–949, Dublin, Ireland, August 23-29 2014. The Impact of Deep Hierarchical Discourse Structures in the Evaluation of Text Coherence Vanessa Wei Feng1, Ziheng Lin2, and Graeme Hirst1 1 Department of Computer Science University of Toronto {weifeng, gh}@cs.toronto.edu 2 Singapore Press Holdings linziheng@gmail.com Abstract Previous work by Lin et al. (2011) demonstrated the effectiveness of using discourse relations for evaluating text coherence. However, their work was based on discourse relations annotated in accordance with the Penn Discourse Treebank (PDTB) (Prasad et al., 2008), which encodes only very shallow discourse structures; therefore, they cannot capture long-distance discourse dependencies. In this paper, we study the impact of deep discourse structures for the task of co- herence evaluation, using two approaches: (1) We compare a model with features derived from discourse relations in the style of Rhetorical Structure Theory (RST) (Mann and Thompson, 1988), which annotate the full hierarchical discourse structure, against our re-implementation of Lin et al.’s model; (2) We compare a model encoded using only shallow RST-style discourse relations, against the one encoded using the complete set of RST-style discourse relations. With an evaluation on two tasks, we show that deep discourse structures are truly useful for better dif- ferentiation of text coherence, and in general, RST-style encoding is more powerful than PDTB- style encoding in these settings. 1 Introduction In a well-written text, utterances are not simply presented in an arbitrary order; rather, they are presented in a logical and coherent form, so that the readers can easily interpret the meaning that the writer wishes to present. Therefore, coherence is one of the most essential aspects of text quality. Given its importance, the automatic evaluation of text coherence is one of the crucial components of many NLP applications. A particularly popular model for the evaluation of text coherence is the entity-based local coherence model of Barzilay and Lapata (B&L) (2005; 2008), which extracts mentions of entities in the text, and models local coherence by the transitions, from one sentence to the next, in the grammatical role of each mention. Since the initial publication of this model, a number of extensions have been proposed, the majority of which are focused on enriching the original feature set. However, these enriched feature sets are usually application-specific, i.e., it requires a certain expertise and intuition to conceive good features. In contrast, we seek insights of better feature encoding from a more general problem: discourse parsing (to be introduced in Section 2). Discourse parsing aims to identify the discourse relations held among various discourse units in the text. Therefore, one can expect that discourse parsing provides useful information to the evaluation of text coherence, because, essentially, the existence and the distribution of discourse relations are the basis of the coherence in a text. In fact, there is already evidence showing that discourse relations can help better capture text coher- ence. Lin et al. (2011) use a PDTB-style discourse parser (to be introduced …

Content truncated to 500 words