Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5716 - 5727 December 7-11, 2022 ©2022 Association for Computational Linguistics Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency Yanzhu Guo1, Chloé Clavel2, Moussa Kamal Eddine1, Michalis Vazirgiannis1 1LIX, École Polytechnique, Institut Polytechnique de Paris, France 2LTCI, Télécom-Paris, Institut Polytechnique de Paris, France {yanzhu.guo, moussa.kamal-eddine}@polytechnique.edu chloe.clavel@telecom-paris.fr, mvazirg@lix.polytechnique.fr Abstract The topic of summarization evaluation has recently attracted a surge of attention due to the rapid development of abstractive sum- marization systems. However, the formu- lation of the task is rather ambiguous, nei- ther the linguistic nor the natural language processing community has succeeded in giving a mutually agreed-upon definition. Due to this lack of well-defined formula- tion, a large number of popular abstractive summarization datasets are constructed in a manner that neither guarantees validity nor meets one of the most essential criteria of summarization: factual consistency. In this paper, we address this issue by combin- ing state-of-the-art factual consistency mod- els to identify the problematic instances present in popular summarization datasets. We release SummFC, a filtered summariza- tion dataset with improved factual consis- tency, and demonstrate that models trained on this dataset achieve improved perfor- mance in nearly all quality aspects. We argue that our dataset should become a valid benchmark for developing and evalu- ating summarization systems. 1 Introduction While the revolutionary success of the Trans- former (Vaswani et al., 2017) architecture has drawn a surge of attention to automatic sum- marization, most research has been focused on improving performance metrics of summa- rization models on a set of popular datasets. It is often taken for granted that these datasets provide representative examples of high qual- ity summaries, and that models capable of producing summaries that are similar to the ones of the dataset are superior in the task of automatic summarization. But is this really the case? Numerous works (Fabbri et al., 2021; Huang et al., 2020) have already pointed out deficien- cies in the most widely employed summariza- tion datasets (e.g. CNN/DailyMail (Hermann et al., 2015; Nallapati et al., 2016) and XSUM (Narayan et al., 2018)). These datasets are typically composed of news articles automat- ically extracted from news websites, paired together with a highlight or introduction sen- tence which serves as the summary. However, the qualities we seek in highlights and intro- ductions are fundamentally different from the ones we seek in summaries. It is thus time for the NLP community to take a step back, revisit the formulation of the summarization task and reconsider the appropriateness of currently employed datasets. The goal of automatic summarization is to take an information source, extract content from it and present the most important con- tent to the user in a concise form and in a manner sensitive to the user’s or application’s needs (Mani, 2001). In the scope of this pa- per, we solely focus on generic summariza- tion (Nenkova and McKeown, 2011), where we make few assumptions about the audience or the …

Content truncated to 500 words