Re-evaluating Evaluation in Text Summarization

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9347–9359, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9347 Re-evaluating Evaluation in Text Summarization Manik Bhandari, Pranav Gour, Atabak Ashfaq, Pengfei Liu, Graham Neubig Carnegie Mellon University {mbhandar,pgour,aashfaq,pliu3,gneubig}@cs.cmu.edu Abstract Automated evaluation metrics as a stand-in for manual evaluation are an essential part of the development of text-generation tasks such as text summarization. However, while the field has progressed, our standard metrics have not – for nearly 20 years ROUGE has been the standard evaluation in most summa- rization papers. In this paper, we make an attempt to re-evaluate the evaluation method for text summarization: assessing the reliabil- ity of automatic metrics using top-scoring sys- tem outputs, both abstractive and extractive, on recently popular datasets for both system- level and summary-level evaluation settings. We find that conclusions about evaluation met- rics on older datasets do not necessarily hold on modern datasets and systems. We release a dataset of human judgments that are col- lected from 25 top-scoring neural summariza- tion systems (14 abstractive and 11 extractive): https://github.com/neulab/REALSumm 1 Introduction In text summarization, manual evaluation, as exem- plified by the Pyramid method (Nenkova and Pas- sonneau, 2004), is the gold-standard in evaluation. However, due to time required and relatively high cost of annotation, the great majority of research papers on summarization use exclusively automatic evaluation metrics, such as ROUGE (Lin, 2004) , JS-2 (Louis and Nenkova, 2013), S3 (Peyrard et al., 2017), BERTScore (Zhang et al., 2020), Mover- Score (Zhao et al., 2019) etc. Among these metrics, ROUGE is by far the most popular, and there is relatively little discussion of how ROUGE may deviate from human judgment and the potential for this deviation to change conclusions drawn re- garding relative merit of baseline and proposed methods. To characterize the relative goodness of evaluation metrics, it is necessary to perform meta- evaluation (Graham, 2015; Lin and Och, 2004), where a dataset annotated with human judgments (e.g. TAC1 2008 (Dang and Owczarzak, 2008)) is used to test the degree to which automatic metrics correlate therewith. However, the classic TAC meta-evaluation datasets are now 6-12 years old2 and it is not clear whether conclusions found there will hold with modern systems and summarization tasks. Two ear- lier works exemplify this disconnect: (1) Peyrard (2019) observed that the human-annotated sum- maries in the TAC dataset are mostly of lower qual- ity than those produced by modern systems and that various automated evaluation metrics strongly disagree in the higher-scoring range in which cur- rent systems now operate. (2) Rankel et al. (2013) observed that the correlation between ROUGE and human judgments in the TAC dataset decreases when looking at the best systems only, even for systems from eight years ago, which are far from today’s state-of-the-art. Constrained by few existing human judgment datasets, it remains unknown how existing metrics behave on current top-scoring summarization sys- tems. In this paper, we ask the question: does the rapid progress of model development in summa- …

Content truncated to 500 words