A Cross-Domain Transferable Neural Coherence Model

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 678–687 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 678 A Cross-Domain Transferable Neural Coherence Model Peng Xu1, Hamidreza Saghir1, Jin Sung Kang1, Teng Long1, Avishek Joey Bose∗,1,2, Yanshuai Cao1, Jackie Chi Kit Cheung1,2,3 1Borealis AI {peng.z.xu,hamidreza.saghir,jinsung.kang,leo.long,yanshuai.cao}@borealisai.com 2McGill University {joey.bose,jcheung}@cs.mcgill.ca 3Canada CIFAR Chair, Mila Abstract Coherence is an important aspect of text qual- ity and is crucial for ensuring its readabil- ity. One important limitation of existing co- herence models is that training on one do- main does not easily generalize to unseen cat- egories of text. Previous work (Li and Ju- rafsky, 2017) advocates for generative models for cross-domain generalization, because for discriminative models, the space of incoher- ent sentence orderings to discriminate against during training is prohibitively large. In this work, we propose a local discriminative neu- ral model with a much smaller negative sam- pling space that can efficiently learn against incorrect orderings. The proposed coherence model is simple in structure, yet it significantly outperforms previous state-of-art methods on a standard benchmark dataset on the Wall Street Journal corpus, as well as in multiple new challenging settings of transfer to unseen cate- gories of discourse on Wikipedia articles. 1 Introduction Coherence is a discourse property that is con- cerned with the logical and semantic organiza- tion of a passage, such that the overall meaning of the passage is expressed fluidly and clearly. It is an important quality measure for text gen- erated by humans or machines, and modelling coherence can benefit many applications, includ- ing summarization, question answering (Verberne et al., 2007), essay scoring (Miltsakaki and Ku- kich, 2004; Burstein et al., 2010) and text gener- ation (Park and Kim, 2015; Kiddon et al., 2016; Holtzman et al., 2018). The ability to generalize to new domains of text is desirable for NLP models in general. Besides the practical reason of avoiding costly retraining ∗Work done while the author was an intern at Borealis AI. on every new domain, for coherence modelling, we would also like our model to make decisions based on the semantic relationships between sen- tences, rather than simply overfit to the structural cues of a specific domain. The standard task used to test a coherence model in NLP is sentence ordering, for example, to distinguish between a coherently ordered list of sentences and a random permutation thereof. Ear- lier work focused on feature engineering, drawing on theories such as Centering Theory (Grosz et al., 1995) and Rhetorical Structure Theory (Thomp- son and Mann, 1987) to propose features based on local entity and lexical transitions, as well as more global concerns regarding topic transitions (Elsner et al., 2007). With the popularization of deep learning, the focus has shifted towards spec- ifying model architectures, including a number of recent models that rely on distributed word rep- resentations used in a deep neural network (Li and Jurafsky, 2017; Nguyen and Joty, 2017; Lo- geswaran et al., 2018). One key …

Content truncated to 500 words