A Unified Neural Coherence Model

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2262–2272, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2262 A Unified Neural Coherence Model Han Cheol Moon∗¶, Tasnim Mohiuddin∗¶, Shafiq Joty∗¶§, and Xu Chi† ¶Nanyang Technological University, Singapore §Salesforce Research Asia, Singapore †A*STAR, Singapore ¶{hancheol001@e., mohi0004@e.}ntu.edu.sg §sjoty@salesforce.com †cxu@simtech.a-star.edu.sg Abstract Recently, neural approaches to coherence modeling have achieved state-of-the-art results in several evaluation tasks. However, we show that most of these models often fail on harder tasks with more realistic application scenar- ios. In particular, the existing models under- perform on tasks that require the model to be sensitive to local contexts such as candidate ranking in conversational dialogue and in ma- chine translation. In this paper, we propose a unified coherence model that incorporates sen- tence grammar, inter-sentence coherence re- lations, and global coherence patterns into a common neural framework. With extensive experiments on local and global discrimina- tion tasks, we demonstrate that our proposed model outperforms existing models by a good margin, and establish a new state-of-the-art. 1 Introduction Coherence modeling involves building text anal- ysis models that can distinguish a coherent text from incoherent ones. It has been a key problem in discourse analysis with applications in text gen- eration, summarization, and coherence scoring. Various linguistic theories have been proposed to formulate coherence, some of which have in- spired development of many of the existing co- herence models. These include the entity-based local models (Barzilay and Lapata, 2008; Elsner and Charniak, 2011b) that consider syntactic real- ization of entities in adjacent sentences, inspired by the Centering Theory (Grosz et al., 1995). An- other line of research uses discourse relations be- tween sentences to predict local coherence (Pitler and Nenkova, 2008; Lin et al., 2011). These meth- ods are inspired by the discourse structure theo- ries like Rhetorical Structure Theory (RST) (Mann and Thompson, 1988) that formalizes coherence in ∗*Equal contribution terms of discourse relations. Other notable meth- ods include word co-occurrence based local mod- els (Soricut and Marcu, 2006), content (or topic distribution) based global models (Barzilay and Lee, 2004), and syntax based local and global models (Louis and Nenkova, 2012). With the neural invasion, some of the above tra- ditional models have got their neural versions with much improved performance. For example, Li and Hovy (2014) implicitly model syntax and inter- sentence relations using a neural framework that uses a recurrent (or recursive) layer to encode each sentence and a fully-connected layer with sigmoid activations to estimate coherence probability for every window of three sentences. Li and Jurafsky (2017) incorporate global topic information with an encoder-decoder architecture, which is also ca- pable of generating discourse. Mesgar and Strube (2018) model change patterns of salient seman- tic information between sentences. Nguyen and Joty (2017); Mohiuddin et al. (2018) propose neu- ral entity grid models using convolutions over dis- tributed representations of entity transitions, and report state-of-the-art results in standard evalua- tion …

Content truncated to 500 words