Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2153–2174 July 9-14, 2023 ©2023 Association for Computational Linguistics Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization Shiyue Zhang∗ David Wan∗ UNC Chapel HillMohit Bansal {shiyue, davidwan, mbansal}@cs.unc.edu Abstract The problems of unfaithful summaries have been widely discussed under the context of abstractive summarization. Though extractive summarization is less prone to the common unfaithfulness issues of abstractive summaries, does that mean extractive is equal to faithful? Turns out that the answer is no. In this work, we define a typology with five types of broad unfaithfulness problems (including and beyond not-entailment) that can appear in extractive summaries, including incorrect coreference, in- complete coreference, incorrect discourse, in- complete discourse, as well as other mislead- ing information. We ask humans to label these problems out of 1600 English summaries pro- duced by 16 diverse extractive systems. We find that 30% of the summaries have at least one of the five issues. To automatically detect these problems, we find that 5 existing faithfulness evaluation metrics for summarization have poor correlations with human judgment. To remedy this, we propose a new metric, EXTEVAL, that is designed for detecting unfaithful extractive summaries and is shown to have the best per- formance. We hope our work can increase the awareness of unfaithfulness problems in ex- tractive summarization and help future work to evaluate and resolve these issues.1 1 Introduction Text summarization is the process of distilling the most important information from a source to pro- duce an abridged version for a particular user or task (Maybury, 1999). Although there are many types of text summarization tasks, in this work, we focus on the task of general purpose single docu- ment summarization. To produce summaries, usu- ally either extractive summarization methods, i.e., extracting sentences from the source, or abstractive ∗ Equal contribution. 1Our data and code are publicly available at https: //github.com/ZhangShiyue/extractive_is_ not_faithful. summarization methods, i.e., generating novel text, are applied (Saggion and Poibeau, 2013). Abstractive summarization attracts more atten- tion from recent works because it can produce more coherent summaries and behaves more like humans (Cohn and Lapata, 2008). Impressive progress has been made for abstractive summarization by large-scale pre-trained models (Lewis et al., 2020; Zhang et al., 2020a). However, unfaithfulness prob- lems, i.e., hallucinating new information or gener- ating content that contradicts the source, are widely spread across models and tasks (Cao et al., 2018; Maynez et al., 2020). Although these problems do not necessarily get captured by typically-used evaluation metrics, e.g., ROUGE (Lin, 2004), even minor unfaithfulness can be catastrophic and drive users away from real-world applications. There- fore, an increasing volume of research has fo- cused on analyzing (Falke et al., 2019; Maynez et al., 2020; Goyal and Durrett, 2021), evaluating (Kryscinski et al., 2020; Goyal and Durrett, 2021; Wang et al., 2020a; Durmus et al., 2020; Scialom et al., 2021; Xie et al., 2021), or addressing (Cao et al., 2018; …

Content truncated to 500 words