Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE?

Findings of the Association for Computational Linguistics: ACL 2022, pages 1547 - 1560 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE? Mousumi Akter, Naman Bansal, Shubhra Kanti Karmaker Santu BDI Lab, Auburn University, Alabama, USA {mza0170, nzb0040, sks0086}@auburn.edu Abstract It has been the norm for a long time to evalu- ate automated summarization tasks using the popular ROUGE metric. Although several studies in the past have highlighted the limi- tations of ROUGE, researchers have struggled to reach a consensus on a better alternative until today. One major limitation of the tra- ditional ROUGE metric is the lack of seman- tic understanding (relies on direct overlap of n-grams). In this paper, we exclusively focus on the extractive summarization task and pro- pose a semantic-aware nCG (normalized cu- mulative gain)-based evaluation metric (called Sem-nCG) for evaluating this task. One fun- damental contribution of the paper is that it demonstrates how we can generate more reli- able semantic-aware ground truths for evalu- ating extractive summarization tasks without any additional human intervention. To the best of our knowledge, this work is the first of its kind. We have conducted extensive experi- ments with this new metric using the widely used CNN/DailyMail dataset. Experimental results show that the new Sem-nCG metric is indeed semantic-aware, shows higher correla- tion with human judgement (more reliable) and yields a large number of disagreements with the original ROUGE metric (suggesting that ROUGE often leads to inaccurate conclu- sions also verified by humans). 1 Introduction Text summarization is a difficult NLP task and an automatic evaluation of this task is even more chal- lenging. However, automatic evaluation is vital for large-scale experiments as it acts as a replacement for time consuming and pricey human evaluation. As such, the reliability and robustness of automatic evaluation is very crucial. The most commonly used metric for evaluating text summarization is ROUGE (Lin, 2004). Al- though ROUGE has been criticized for considering direct lexical overlap and thus not being semantic- aware, the majority of summarization models’ as- sessments today are still based on ROUGE scores. In this paper, we revisit the popular ROUGE metric exclusively in the context of evaluating extractive summarization task, a task where phrases and sen- tences from the original text are extracted to create a summary. As such, if the human-written sum- mary includes more novel words than the original document, ROUGE will provide a poor score to extractive summaries due to a lack of semantic awareness. Another limitation of the ROUGE met- ric in the context of extractive summarization is the following: while the extractive summarization task is generally framed as a sentence ranking problem, the ROUGE metric was not originally proposed for evaluating the quality of a ranker. Indeed, the heavily used technique behind extractive summa- rization is to rank sentences from the original docu- ment according to how well they reflect the overall description and then create a summary by concate- nating the top-ranked …

Content truncated to 500 words