How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation

Proceedings of the 29th International Conference on Computational Linguistics, pages 6035–6049 October 12–17, 2022. 6035 How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation Julius Steen Katja Markert Department of Computational Linguistics Heidelberg University 69120 Heidelberg, Germany (steen|markert)@cl.uni-heidelberg.de Abstract Automatically evaluating the coherence of sum- maries is of great significance both to enable cost-efficient summarizer evaluation and as a tool for improving coherence by selecting high- scoring candidate summaries. While many different approaches have been suggested to model summary coherence, they are often eval- uated using disparate datasets and metrics. This makes it difficult to understand their relative performance and identify ways forward to- wards better summary coherence modelling. In this work, we conduct a large-scale investiga- tion of various methods for summary coherence modelling on an even playing field. Addition- ally, we introduce two novel analysis measures, intra-system correlation and bias matrices, that help identify biases in coherence measures and provide robustness against system-level con- founders. While none of the currently available automatic coherence measures are able to as- sign reliable coherence scores to system sum- maries across all evaluation metrics, large-scale language models fine-tuned on self-supervised tasks show promising results, as long as fine- tuning takes into account that they need to gen- eralize across different summary lengths. 1 Introduction Automatically generated summaries should not only be informative, but also well-written and co- herent. While informativeness is routinely evalu- ated automatically with ROUGE (Lin, 2004), there is no agreement on how to evaluate summary co- herence. However, automatic evaluation is highly desirable to reduce evaluation costs and as a tool for improving summarizer output, e.g. as reranker. Many coherence measures (CMs) have been suggested for automatically assigning a coher- ence score to a summary, including learning from human judgements (Barzilay and Lapata, 2008; Tien Nguyen and Joty, 2017; Xenouleas et al., 2019; Mesgar et al., 2021), learning from the shuf- fle task (Mohiuddin et al., 2021; Jwalapuram et al., 2022), where models are trained to discriminate original documents from documents with random- ized sentence order (Barzilay and Lapata, 2008), us- ing next sentence prediction as a proxy task (Koto et al., 2022), and finally unsupervised measures that exploit heuristics (Pitler et al., 2010; Zhu and Bhat, 2020) or large-scale LMs (Yuan et al., 2021). CM performance is then evaluated by comparing the automatic scores to human coherence scores on a set of system summaries. However, this evaluation is often conducted on disparate datasets. It also often uses system out- puts from DUC conferences (Barzilay and Lap- ata, 2008; Tien Nguyen and Joty, 2017; Xenouleas et al., 2019; Mesgar et al., 2021), which do not necessarily represent recent advances in text sum- marizers. In addition, there is no agreement on how the CM scores should be compared to human scores. System-level correlation (Xenouleas et al., 2019; Fabbri et al., 2021), pairwise ranking accu- racy (Barzilay and Lapata, 2008; Tien Nguyen and Joty, 2017; Mesgar et al., 2021) and summary-level correlation …

Content truncated to 500 words