Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6044 - 6059 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling Prathyusha Jwalapuram† and Shafiq Joty†§ and Xiang Lin† †Nanyang Technological University, Singapore §Salesforce Research Asia, Singapore {jwal0001,srjoty,linx0057}@ntu.edu.sg Abstract Given the claims of improved text generation quality across various pre-trained neural mod- els, we consider the coherence evaluation of machine generated text to be one of the prin- cipal applications of coherence models that needs to be investigated. Prior work in neu- ral coherence modeling has primarily focused on devising new architectures for solving the permuted document task. We instead use a basic model architecture and show significant improvements over state of the art within the same training regime. We then design a harder self-supervision objective by increasing the ra- tio of negative samples within a contrastive learning setup, and enhance the model further through automatic hard negative mining cou- pled with a large global negative queue encoded by a momentum encoder. We show empiri- cally that increasing the density of negative samples improves the basic model, and using a global negative queue further improves and stabilizes the model while training with hard negative samples. We evaluate the coherence model on task-independent test sets that resem- ble real-world applications and show signifi- cant improvements in coherence evaluations of downstream tasks.1 1 Introduction Coherence is a property of a well-written text that makes it different from a random set of sentences: sentences in a coherent text are connected in sys- tematic ways such that each sentence follows nat- urally from previous ones and leads into the fol- lowing ones (Halliday and Hasan, 1976; Grosz and Sidner, 1986). Coherence models (Barzilay and Lapata, 2005) that can distinguish a coherent text from incoherent ones have a wide range of applica- tions in language generation, summarization, and coherence assessment tasks such as essay scoring and sentence ordering. 1Our code and data are available at https://ntunlpsg.github.io/project/coherence-paradigm With recent advancements in neural methods, claims of fluency in summarization (Liu et al., 2017; Celikyilmaz et al., 2018), language modeling (Radford et al., 2019; Brown et al., 2020), response generation (Zhang et al., 2020; Hosseini-Asl et al., 2020) and human parity in machine translation (Hassan et al., 2018) have led to calls for finer- grained discourse-level evaluations (Läubli et al., 2018; Sharma et al., 2019; Popel et al., 2020), since traditional metrics such as BLEU and ROUGE are unable to measure text quality and readabil- ity (Paulus et al., 2018; Reiter, 2018). Coherence models that can evaluate machine-generated text have become the need of the hour. A majority of coherence models proposed op- timize their learning objectives on the permuted document task using the Penn Treebank (WSJ) cor- pus. An original article is considered a ‘positive’ sample of a coherent document, while a permuta- tion of its sentences is considered a ‘negative’ or incoherent sample (see Appendix A.1 for an ex- ample). Models are …

Content truncated to 500 words