Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3528–3539 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3528 Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks Tasnim Mohiuddin∗¶, Prathyusha Jwalapuram∗¶, Xiang Lin∗¶, and Shafiq Joty ∗¶† ¶Nanyang Technological University, Singapore †Salesforce Research Asia, Singapore {mohi0004, jwal0001, linx0057, srjoty}@ntu.edu.sg Abstract Although coherence modeling has come a long way in developing novel models, their evalu- ation on downstream applications for which they are purportedly developed has largely been neglected. With the advancements made by neural approaches in applications such as machine translation (MT), summarization and dialog systems, the need for coherence evalu- ation of these tasks is now more crucial than ever. However, coherence models are typically evaluated only on synthetic tasks, which may not be representative of their performance in downstream applications. To investigate how representative the synthetic tasks are of down- stream use cases, we conduct experiments on benchmarking well-known traditional and neu- ral coherence models on synthetic sentence or- dering tasks, and contrast this with their perfor- mance on three downstream applications: co- herence evaluation for MT and summarization, and next utterance prediction in retrieval-based dialog. Our results demonstrate a weak cor- relation between the model performances in the synthetic tasks and the downstream appli- cations, motivating alternate training and eval- uation methods for coherence models.1 1 Introduction and Related Work Coherence is an important aspect of discourse that distinguishes a well-written text from a poorly- written one that is difficult to comprehend (Halliday and Hasan, 1976). Computational models that can assess coherence have applications in text genera- tion and ranking, such as summarization, machine translation, essay scoring and dialog systems. Researchers have proposed a number of formal theories of discourse coherence, which have in- ∗*Equal contribution 1Code and data used for evaluation available at https://ntunlpsg.github.io/project/ coherence/coh-eval/ spired the development of many coherence mod- els – both traditional and neural ones. Inspired by the Centering Theory (Grosz et al., 1995), the entity based local models (Barzilay and Lapata, 2008; Elsner and Charniak, 2011b) formulate co- herence in terms of syntactic roles (e.g., subject, object) of entities in nearby sentences. Another branch of models (Pitler and Nenkova, 2008; Lin et al., 2011; Feng et al., 2014) use coherence re- lations between adjacent sentences to model local coherence, inspired by the discourse structure the- ories of Mann and Thompson (1988) and Webber (2004). Other traditional methods include word co- occurrence based local models (Soricut and Marcu, 2006), topic based global models (Barzilay and Lee, 2004; Elsner et al., 2007), and syntax based local and global models (Louis and Nenkova, 2012). Despite continuous research efforts in develop- ing novel coherence models, their usefulness in downstream applications has largely been ignored. They have been evaluated in mainly two ways. The most common approach has been to evaluate them on synthetic discrimination tasks that involve iden- tifying the right order of the sentences at the lo- cal and global levels (Barzilay and Lapata, 2008; Elsner …

Content truncated to 500 words