Automatically Evaluating Text Coherence Using Discourse Relations

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 997–1006, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Automatically Evaluating Text Coherence Using Discourse Relations Ziheng Lin, Hwee Tou Ng and Min-Yen Kan Department of Computer Science National University of Singapore 13 Computing Drive Singapore 117417 {linzihen,nght,kanmy}@comp.nus.edu.sg Abstract We present a novel model to represent and assess the discourse coherence of text. Our model assumes that coherent text implicitly favors certain types of discourse relation tran- sitions. We implement this model and apply it towards the text ordering ranking task, which aims to discern an original text from a per- muted ordering of its sentences. The experi- mental results demonstrate that our model is able to significantly outperform the state-of- the-art coherence model by Barzilay and Lap- ata (2005), reducing the error rate of the previ- ous approach by an average of 29% over three data sets against human upper bounds. We fur- ther show that our model is synergistic with the previous approach, demonstrating an error reduction of 73% when the features from both models are combined for the task. 1 Introduction The coherence of a text is usually reflected by its dis- course structure and relations. In Rhetorical Struc- ture Theory (RST), Mann and Thompson (1988) ob- served that certain RST relations tend to favor one of two possible canonical orderings. Some rela- tions (e.g., Concessive and Conditional) favor ar- ranging their satellite span before the nucleus span. In contrast, other relations (e.g., Elaboration and Ev- idence) usually order their nucleus before the satel- lite. If a text that uses non-canonical relation order- ings is rewritten to use canonical orderings, it often improves text quality and coherence. This notion of preferential ordering of discourse relations is observed in natural language in general, and generalizes to other discourse frameworks aside from RST. The following example shows a Contrast relation between the two sentences. (1) [ Everyone agrees that most of the nation’s old bridges need to be repaired or replaced. ]S1 [ But there’s disagreement over how to do it. ]S2 Here the second sentence provides contrasting infor- mation to the first. If this order is violated without rewording (i.e., if the two sentences are swapped), it produces an incoherent text (Marcu, 1996). In addition to the intra-relation ordering, such preferences also extend to inter-relation ordering: (2) [ The Constitution does not expressly give the president such power. ]S1 [ However, the president does have a duty not to violate the Constitution. ]S2 [ The question is whether his only means of defense is the veto. ]S3 The second sentence above provides a contrast to the previous sentence and an explanation for the next one. This pattern of Contrast-followed-by-Cause is rather common in text (Pitler et al., 2008). Ordering the three sentences differently results in incoherent, cryptic text. Thus coherent text exhibits measurable prefer- ences for specific intra- and inter-discourse relation ordering. Our key idea is to use the converse of this phenomenon …

Content truncated to 500 words