Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 128–137, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE Yvette Graham ADAPT Centre School of Computer Science and Statistics Trinity College Dublin graham.yvette@gmail.com Abstract We provide an analysis of current evalua- tion methodologies applied to summariza- tion metrics and identify the following ar- eas of concern: (1) movement away from evaluation by correlation with human as- sessment; (2) omission of important com- ponents of human assessment from eval- uations, in addition to large numbers of metric variants; (3) absence of methods of significance testing improvements over a baseline. We outline an evaluation methodology that overcomes all such chal- lenges, providing the first method of sig- nificance testing suitable for evaluation of summarization metrics. Our evaluation re- veals for the first time which metric vari- ants significantly outperform others, op- timal metric variants distinct from cur- rent recommended best variants, as well as machine translation metric BLEU to have performance on-par with ROUGE for the purpose of evaluation of summariza- tion systems. We subsequently replicate a recent large-scale evaluation that relied on, what we now know to be, suboptimal ROUGE variants revealing distinct conclu- sions about the relative performance of state-of-the-art summarization systems. 1 Introduction Automatic metrics of summarization evaluation have their origins in machine translation (MT), with ROUGE (Lin and Hovy, 2003), the first and still most widely used automatic summarization metric, comprising an adaption of the BLEU score (Papineni et al., 2002). Automatic evaluation in MT and summarization have much in common, as both involve the automatic comparison of system- generated texts with one or more human-generated reference texts, contrasting either system-output translations or peer summaries with human ref- erence translations or model summaries, depend- ing on the task. In both MT and summarization evaluation, any newly proposed automatic metric must be assessed by the degree to which it pro- vides a good substitute of human assessment, and although there are obvious parallels between eval- uation of systems in the two areas, when it comes to evaluation of metrics, summarization has di- verged considerably from methodologies applied to evaluation of metrics in MT. Since the inception of BLEU, evaluation of au- tomatic metrics in MT has been by correlation with human assessment. In contrast in summa- rization, over the years since the introduction of ROUGE, summarization evaluation has seen a va- riety of different methodologies applied to evalu- ation of its metrics. Evaluation of summarization metrics has included, for example, the ability of a metric/significance test combination to distinguish between sets of human and system-generated sum- maries (Rankel et al., 2011), or by accuracy of conclusions drawn from metrics when combined with a particular significance test, Wilcoxon rank- sum (Owczarzak et al., 2012). Besides moving away from well-established methods such as correlation with human judg- ment, previous summarization metric evaluations have been additionally limited by inclusion of only a small proportion of possible metrics …

Content truncated to 500 words