Evaluating Text Coherence at Sentence and Paragraph Levels

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1695–1703 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1695 Evaluating Text Coherence at Sentence and Paragraph Levels Sennan Liu13, Shuang Zeng23, Sujian Li3 1Department of English and International Study, China Foreign Affairs University, Beijing, China 2School of Software and Microelectronics, Peking University, Beijing, China 3MOE Key Lab of Computational Linguistics, Peking University, Beijing, China stan.x.liu.18@gmail.com, {zengs,lisujian}@pku.edu.cn Abstract In this paper, to evaluate text coherence, we propose the paragraph ordering task as well as conducting sentence ordering. We collected four distinct corpora from different domains on which we investigate the adaptation of existing sentence ordering methods to a paragraph ordering task. We also compare the learnability and robustness of existing models by artificially creating mini datasets and noisy datasets respectively and verifying the efficiency of established models under these circumstances. Furthermore, we carry out human evaluation on the rearranged passages from two competitive models and confirm that WLCS-l is a better metric performing significantly higher correlations with human rating than τ, the most prevalent metric used before. Results from these evaluations show that except for certain extreme conditions, the recurrent graph neural network-based model is an optimal choice for coherence modeling. Keywords: text ordering, coherence modeling, learnability verification, robustness assessment, human evaluation 1. Introduction Coherence modeling has been a topic for discourse analysis for a long time (Lapata, 2003). Text ordering is the standard task used to test a coherence model in NLP. While earlier work aims at distinguishing between a coherently ordered list of sentences and a random permutation thereof, recent studies attempt to generate a consecutive text from a set of given sentences. Various frameworks exist, focusing on linguistic features via statistical models (Lapata, 2003; Barzilay and Lee, 2004; Barzilay and Lapata, 2005; Elsner and Charniak, 2011). Especially, entity based models (Barzilay and La- pata, 2008; Guinaudeau and Strube, 2013) have shown the effectiveness of exploiting entities for this task. With the popularization of neural networks, studies formulate the problem as neural pair-wise discrimination (Li and Hovy, 2014; Chen et al., 2016), with a purpose on training a dis- criminator to separate coherent clique or pairs from non- coherent ones. Recently, thanks to the success of set- to-sequence framework (Vinyals et al., 2015; Vinyals et al., 2016), the focus shifts to generative ordering problem (Gong et al., 2016; Logeswaran et al., 2017; Cui et al., 2018; Yin et al., 2019), aiming at finding a framework to learn to generate an optimal order from a bunch of in- put sequences. However, most of these studies only focus on the sentence level task and experiment using the ab- stracts from academic papers, which is unrealistic in the real world where various lengths and text domains may be used. Moreover, none of them analyze the adaptability of models with respect to different qualities of language re- source, which prevents further study to improve the models in a fine-grained aspect. In this paper, we …

Content truncated to 500 words