Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics

Proceedings of the 28th International Conference on Computational Linguistics, pages 5702–5711 Barcelona, Spain (Online), December 8-13, 2020 5702 Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics Manik Bhandari Pranav Gour Atabak Ashfaq Pengfei Liu Carnegie Mellon University {mbhandar,pgour,aashfaq,pliu3}@cs.cmu.edu Abstract In text summarization, evaluating the efficacy of automatic metrics without human judgments has become recently popular. One exemplar work (Peyrard, 2019) concludes that automatic metrics strongly disagree when ranking high-scoring summaries. In this paper, we revisit their experiments and find that their observations stem from the fact that metrics disagree in ranking summaries from any narrow scoring range. We hypothesize that this may be because sum- maries are similar to each other in a narrow scoring range and are thus, difficult to rank. Apart from the width of the scoring range of summaries, we analyze three other properties that impact inter-metric agreement - Ease of Summarization, Abstractiveness, and Coverage. To encourage reproducible research, we make all our analysis code and data publicly available.1 1 Introduction Automatic metrics play a significant role in summarization evaluation, profoundly affecting the direction of system optimization. Due to its importance, evaluating the quality of evaluation metrics, also known as meta-evaluation has been a crucial step. Generally, there are two meta-evaluation strategies: (i) assessing how well each metric correlates with human judgments (Lin, 2004; Ng and Abrecht, 2015; Louis and Nenkova, 2013; Peyrard et al., 2017; Bhandari et al., 2020), which requires procuring manual annotations that are expensive and time-consuming, and (ii) measuring the correlation between different metrics (Peyrard, 2019), which is a human judgment-free method. In this work, we focus on the latter and ask two research questions: RQ1: How do automated metrics correlate when ranking summaries in different scoring ranges (low, average, and high)? We revisit the experiments of Peyrard (2019) which concludes that automated met- rics strongly disagree for ranking high-scoring summaries. 2 We find that the scoring range has little effect on the correlation of metrics. It is rather the width of the scoring range which affects inter-metric correlation. Specifically, we observe that metrics agree in ranking summaries from the full scoring range but disagree in ranking summaries from low, average, and high scoring ranges when taken separately. RQ2: Which other factors affect the correlations of metrics? In addition to the width of the scoring range, we analyze three properties of a reference summary on inter-metric correlation - Ease of Summa- rization, Abstractiveness and Coverage. Overall we find that for highly extractive document-reference summary pairs, inter-metric correlation is high whereas metrics disagree when ranking summaries of abstractive document-reference summary pairs. We summarize our contributions as follows: (1) We extend the analysis of Peyrard (2019) and find that not only do metrics disagree in the high scoring range, they also disagree in the low and medium scoring range. (2) We perform our analysis on the popular CNN/Dailymail dataset using traditional lexical 1https://github.com/manikbhandari/RevisitSummEvalMetrics 2Peyrard (2019) uses three experiments to reach their conclusion. Due to limitations of space, we focus on the …

Content truncated to 500 words