SummEval: Re-evaluating Summarization Evaluation

SummEval: Re-evaluating Summarization Evaluation Alexander R. Fabbri†∗ Wojciech Kry´sci´nski‡ ∗ Bryan McCann‡ Caiming Xiong‡ Richard Socher‡ Dragomir Radev† ‡ †Yale University ‡Salesforce Research {alexander.fabbri,dragomir.radev}@yale.edu, {kryscinski,cxiong}@salesforce.com richard@socher.org bryan.mccann.is@gmail.com Abstract The scarcity of comprehensive up-to-date studies on evaluation metrics for text summari- zation and the lack of consensus regarding evaluation protocols continue to inhibit pro- gress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd- sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model- generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evalua- tion protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments. 1 Introduction Text summarization aims to compress long doc- ument(s) into a short, fluent, and human-readable form that preserves the most salient information from the source document. ∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et al., 2018; Narayan et al., 2018). Recent advances in pretrained language models, such as BERT (Devlin et al., 2019), have motivated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al., 2019b; Dong et al., 2019; Ziegler et al., 2019; Raffel et al., 2019; Lewis et al., 2019). A standard dataset for training summarization models is the CNN/DailyMail corpus (Hermann et al., 2015), originally a question answering task, which was repurposed for summarization by Nallapati et al. (2016). The dataset consists of news articles and associated human-created bullet- point summaries. The ROUGE (Lin, 2004b) metric, which measures lexical overlap between generated and target summaries, is then typically used together with crowd-sourced human annota- tions for model evaluation. While the currentsetup has become standardized, we believe several fac- tors prevent a more complete comparison of mod- els, thus negatively impacting the progress of the field. As noted by Hardy et al. (2019), recent papers vastly differ in their evaluation protocol. Existing work often limits model comparisons to only a few baselines and offers human evaluations which are largely inconsistent with prior work. Additionally, despite problems associated with ROUGE when used outside of its original setting (Liu and Liu, 2008; …

Content truncated to 500 words