A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods

A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods Daniel Deutsch, Rotem Dror, and Dan Roth Department of Computer and Information Science University of Pennsylvania, USA {ddeutsch,rtmdrr,danroth}@seas.upenn.edu Abstract The quality of a summarization evaluation metric is quantified by calculating the correla- tion between its scores and human annotations across a large number of summaries. Currently, it is unclear how precise these correlation es- timates are, nor whether differences between two metrics’ correlations reflect a true differ- ence or if it is due to mere chance. In this work, we address these two problems by proposing methods for calculating confidence intervals and running hypothesis tests for correlations using two resampling methods, bootstrapping and permutation. After evaluating which of the proposed methods is most appropriate for summarization through two simulation exper- iments, we analyze the results of applying these methods to several different automatic evaluation metrics across three sets of human annotations. We find that the confidence in- tervals are rather wide, demonstrating high uncertainty in the reliability of automatic met- rics. Further, although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do so in some evaluation settings.1 1 Introduction Accurately estimating the quality of a summary is critical for understanding whether one summa- rization model produces better summaries than another. Because manually annotating summary quality is costly and time consuming, researchers have developed automatic metrics that approx- imate human judgments (Lin, 2004; Tratz and Hovy, 2008; Giannakopoulos et al., 2008; Zhao et al., 2019; Deutsch et al., 2021, among others). Currently, automatic metrics themselves are evaluated by calculating the correlations between 1Our code is available at https://github.com /CogComp/stat-analysis-experiments. their scores and human-annotated quality scores. The value of a metric’s correlation represents how similar its scores are to humans’, and one metric is said to be a better approximation of human judgments than another if its correlation is higher. However, there is no standard practice in sum- marization for calculating confidence intervals (CIs) for the correlation values or running hypoth- esis tests on the difference between two metrics’ correlations. This leaves the community in doubt about how effective automatic metrics really are at replicating human judgments as well as whether the difference between two metrics’ correlations is truly reflective of one metric being better than the other or if it is an artifact of random chance. In this work, we propose methods for cal- culating CIs and running hypothesis tests for summarization metrics. After demonstrating the usefulness of our methods through a pair of sim- ulation experiments, we then analyze the results of applying the statistical analyses to a set of summarization metrics and three datasets. The methods we propose are based on the re- sampling techniques of bootstrapping (Efron and Tibshirani, 1993) and permutation (Noreen, 1989). Resampling techniques are advantageous because, unlike parametric methods, they do not make assumptions which are invalid in the case of sum- marization (§3.1; §4.1). Bootstrapping and permu- tation techniques use a subroutine that samples a new dataset …

Content truncated to 500 words