Neural Net Models of Open-domain Discourse Coherence

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 198–209 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Neural Net Models of Open-domain Discourse Coherence Jiwei Li and Dan Jurafsky Computer Science Department Stanford University, Stanford, USA jiweil,jurafsky@stanford.edu Abstract Discourse coherence is strongly associated with text quality, making it important to nat- ural language generation and understand- ing. Yet existing models of coherence focus on measuring individual aspects of coher- ence (lexical overlap, rhetorical structure, entity centering) in narrow domains. In this paper, we describe domain- independent neural models of discourse co- herence that are capable of measuring mul- tiple aspects of coherence in existing sen- tences and can maintain coherence while generating new sentences. We study both discriminative models that learn to distin- guish coherent from incoherent discourse, and generative models that produce coher- ent text, including a novel neural latent- variable Markovian generative model that captures the latent discourse dependencies between sentences in a text. Our work achieves state-of-the-art perfor- mance on multiple coherence evaluations, and marks an initial step in generating co- herent texts given discourse contexts. 1 Introduction Modeling discourse coherence (the way parts of a text are linked into a coherent whole) is essen- tial for summarization (Barzilay and McKeown, 2005), text planning (Hovy, 1988; Marcu, 1997) question-answering (Verberne et al., 2007), and even psychiatric diagnosis (Elvev˚ag et al., 2007; Bedi et al., 2015). Various frameworks exist, each tackling aspects of coherence. Lexical cohesion (Halliday and Hasan, 1976; Morris and Hirst, 1991) models chains of words and synonyms. Psychological mod- els of discourse (Foltz et al., 1998; Foltz, 2007; McNamara et al., 2010) use LSA embeddings to generalize lexical cohesion. Relational models like RST (Mann and Thompson, 1988; Lascarides and Asher, 1991) define relations that hierarchically structure texts. The entity grid model (Barzilay and Lapata, 2008) and its extensions1 capture the referential coherence of entities moving in and out of focus across a text. Each captures only a single aspect of coherence, and all focus on scoring exist- ing sentences, rather than on generating coherent discourse for tasks like abstractive summarization. Here we introduce two classes of neural models for discourse coherence. Our discriminative mod- els induce coherence by treating human generated texts as coherent examples and texts with random sentence replacements as negative examples, feed- ing LSTM sentence embeddings of pairs of consec- utive sentences to a classifier. These achieve state- of-the-art (96% accuracy) on the standard domain- specific sentence-pair-ordering dataset (Barzilay and Lapata, 2008), but suffer in a larger open- domain setting due to the small semantic space that negative sampling is able to cover. Our generative models are based on augument- ing encoder-decoder models with latent variables to model discourse relationships across sentences, including (1) a model that incorporates an HMM- LDA topic model into the generative model and (2) an end-to-end model that introduces a Markov- structured neural latent variable, inspired by re- cent work on training latent-variable recurrent nets (Bowman et al., 2015; Serban …

Content truncated to 500 words