Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 6038 - 6052 July 10-15, 2022 ©2022 Association for Computational Linguistics Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics Daniel Deutsch, Rotem Dror, and Dan Roth Department of Computer and Information Science University of Pennsylvania {ddeutsch,rtmdrr,danroth}@seas.upenn.edu Abstract How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system- level correlations. We identify two ways in which the definition of the system-level cor- relation is inconsistent with how metrics are used to evaluate systems in practice and pro- pose changes to rectify this disconnect. First, we calculate the system score for an automatic metric using the full test set instead of the sub- set of summaries judged by humans, which is currently standard practice. We demonstrate how this small change leads to more precise es- timates of system-level correlations. Second, we propose to calculate correlations only on pairs of systems that are separated by small differences in automatic scores which are com- monly observed in practice. This allows us to demonstrate that our best estimate of the correlation of ROUGE to human judgments is near 0 in realistic scenarios. The results from the analyses point to the need to collect more high-quality human judgments and to improve automatic metrics when differences in system scores are small.1 1 Introduction Automatic evaluation metrics are the most common method that researchers use to quickly and cheaply approximate how humans would rate the quality of a summarization system (Lin, 2004; Louis and Nenkova, 2013; Zhao et al., 2019; Zhang et al., 2020; Deutsch et al., 2021a, among others). The quality of a metric — how similarly it replicates human judgments of systems — is quantified by calculating the correlation between the metric’s scores and human judgments on a set of systems, known as the system-level correlation (Louis and Nenkova, 2013; Deutsch et al., 2021b). Accurately estimating system-level correlations is critically important. Summarization researchers 1Our code is available at http://cogcomp.org/ page/publication_view/973. use automatic metrics during system development to make decisions about which ideas work and which do not, and systems from different research groups are ranked by automatic metrics to define which system is the “state-of-the-art.” If we do not have precise estimates of metric quality, it is not clear how much trust the community should put in such evaluation methodologies. At present, there are disconnects between how automatic metrics are evaluated and how they are used to evaluate systems. First, the metrics’ scores which are used in practice are not the ones which are evaluated in system-level correlations; re- searchers compare systems based on metric scores calculated on the entire test set but calculate scores for system-level correlations when evaluating met- rics on a much smaller subset of judged summaries. Second, metrics are evaluated in a setting that is much easier than how they are actually used. Met- ric correlations are calculated using systems that vary greatly in quality, …

Content truncated to 500 words