How coherent are neural models of coherence?

Proceedings of the 28th International Conference on Computational Linguistics, pages 6126–6138 Barcelona, Spain (Online), December 8-13, 2020 6126 How coherent are neural models of coherence? Leila Pishdad Federico Fancellu Ran Zhang Afsaneh Fazly Samsung AI Centre Toronto (SAIC Toronto) {leila.p, federico.f, ran.zhang, a.fazly}@samsung.com Abstract Despite the recent advances in coherence modelling, most such models including state-of-the-art neural ones, are evaluated on either contrived proxy tasks such as the standard order discrimination benchmark, or tasks that require special expert annotation. Moreover, most evaluations are conducted on small newswire corpora. To address these shortcomings, in this paper we propose four generic evaluation tasks that draw on different aspects of coherence at both the lexical and document levels, and can be applied to any corpora. In designing these tasks, we aim at capturing coherence-specific properties, such as the correct use of discourse connectives, lexical cohesion, as well as the overall temporal and causal consistency among events and participants in a story. Importantly, our proposed tasks either rely on automatically-generated data, or data annotated for other purposes, hence alleviating the need for annotation specifically targeted to the task of coherence modelling. We perform experiments with several existing state-of-the-art neural models of coherence on these tasks, across large corpora from different domains, including newswire, dialogue, as well as narrative and instructional text. Our findings point to a strong need for revisiting the common practices in the development and evaluation of coherence models. 1 Introduction A variety of research has focused on modelling textual coherence, with the goal of distinguishing coherent from incoherent documents. This distinction has important repercussions for a variety of downstream tasks, including automatic essay scoring, document summarization, and natural language generation (Jernite et al., 2017; Li and Jurafsky, 2017; Wu and Hu, 2018). Given the importance of the task, there is a long line of approaches proposed for modelling coherence. Most earlier models leverage linguistic features to discriminate between coherent and incoherent documents: The classic work of Barzilay and Lapata (2008), and several follow-up studies (Elsner and Charniak, 2011; Guinaudeau and Strube, 2013; Zhang et al., 2015) model coherence by using entity transitions — information about how semantic roles change across sentences. Lin et al. (2011) and Feng et al. (2014) rely on discourse relation transitions, whereas Louis and Nenkova (2012) focus on patterns of syntactic transition extracted from a constituency tree. Alternatively, other models have drawn on topic transitions across sentences in a document to assess its coherence (Morris and Hirst, 1991; Somasundaran et al., 2014; Mesgar and Strube, 2016). Recent models on the other hand, rely on neural architectures with little to no feature engineering required. Nguyen and Joty (2017) and Mohiuddin et al. (2018) propose neural variants of the entity transition model of Barzilay and Lapata (2008), while others devise approaches to learning distributed representations of sentences/documents that also encode coherence (Li and Hovy, 2014; Jernite et al., 2017; Li and Jurafsky, 2017; Logeswaran et al., 2018). These specialized encoders often claim to combine various aspects of coherence into …

Content truncated to 500 words