A Model of Coherence Based on Distributed Sentence Representation

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2039–2048, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics A Model of Coherence Based on Distributed Sentence Representation Jiwei Li1 and Eduard Hovy3 1Computer Science Department, Stanford University, Stanford, CA 94305, USA 3Language Technology Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA jiweil@stanford.edu ehovy@andrew.cmu.edu Abstract Coherence is what makes a multi-sentence text meaningful, both logically and syn- tactically. To solve the challenge of or- dering a set of sentences into coherent or- der, existing approaches focus mostly on defining and using sophisticated features to capture the cross-sentence argumenta- tion logic and syntactic relationships. But both argumentation semantics and cross- sentence syntax (such as coreference and tense rules) are very hard to formalize. In this paper, we introduce a neural network model for the coherence task based on distributed sentence representation. The proposed approach learns a syntactico- semantic representation for sentences au- tomatically, using either recurrent or re- cursive neural networks. The architecture obviated the need for feature engineering, and learns sentence representations, which are to some extent able to capture the ‘rules’ governing coherent sentence struc- ture. The proposed approach outperforms existing baselines and generates the state- of-art performance in standard coherence evaluation tasks1. 1 Introduction Coherence is a central aspect in natural language processing of multi-sentence texts. It is essen- tial in generating readable text that the text plan- ner compute which ordering of clauses (or sen- tences; we use them interchangeably in this paper) is likely to support understanding and avoid con- fusion. As Mann and Thompson (1988) define it, A text is coherent when it can be ex- plained what role each clause plays with regard to the whole. 1Code available at stanford.edu/˜jiweil/ or by request from the first author. Several researchers in the 1980s and 1990s ad- dressed the problem, the most influential of which include: Rhetorical Structure Theory (RST; (Mann and Thompson, 1988)), which defined about 25 relations that govern clause interde- pendencies and ordering and give rise to text tree structures; the stepwise assembly of seman- tic graphs to support adductive inference toward the best explanation (Hobbs et al., 1988); Dis- course Representation Theory (DRT; (Lascarides and Asher, 1991)), a formal semantic model of discourse contexts that constrain coreference and quantification scoping; the model of intention- oriented conversation blocks and their stack-based queueing to model attention flow (Grosz and Sid- ner, 1986), and more recently an inventory of a hundred or so binary inter-clause relations and as- sociated annotated corpus (Penn Discourse Tree- bank. Work in text planning implemented some of these models, especially operationalized RST (Hovy, 1988) and explanation relations (Moore and Paris, 1989) to govern the planning of coher- ent paragraphs. Other computational work defined so called schemas (McKeown, 1985), frames with fixed sequences of clause types to achieve stereo- typical communicative intentions. Little of this work survives. Modern research tries simply to order a collection of clauses or sen- tences without giving an account of …

Content truncated to 500 words