Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5935–5941 June 6–11, 2021. ©2021 Association for Computational Linguistics 5935 Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection Sihao Chen1∗ Fan Zhang2 Kazoo Sone2 Dan Roth1∗ 1University of Pennsylvania 2Google {sihaoc, danroth}@cis.upenn.edu, {zhanfan, sone}@google.com Abstract Despite significant progress in neural abstrac- tive summarization, recent studies have shown that the current models are prone to generat- ing summaries that are unfaithful to the orig- inal context. To address the issue, we study contrast candidate generation and selection as a model-agnostic post-processing technique to correct the extrinsic hallucinations (i.e. infor- mation not present in the source text) in un- faithful summaries. We learn a discriminative correction model by generating alternative can- didate summaries where named entities and quantities in the generated summary are re- placed with ones with compatible semantic types from the source document. This model is then used to select the best candidate as the final output summary. Our experiments and analysis across a number of neural summariza- tion systems show that our proposed method is effective in identifying and correcting extrinsic hallucinations. We analyze the typical halluci- nation phenomenon by different types of neu- ral summarization systems, in hope to provide insights for future work on the direction. 1 Introduction Abstractive Summarization is the task of produc- ing a concise and fluent summary that is salient and faithful to the source document(s). Data- driven, neural methods (Rush et al., 2015; Nallapati et al., 2016; See et al., 2017), and the more recent, pretrained transformer language models (Vaswani et al., 2017; Devlin et al., 2019; Liu and Lapata, 2019), have shown improvements in the fluency and salience of generated summaries. However, less progress has been made on im- proving the faithfulness of the generated sum- maries, that is, producing a summary that is en- tailed by the information presented in the source document. Despite the increased level of perfor- mance under automatic metrics such as ROUGE ∗ Most of the work done while the authors were at Google. Source: He was re-elected for a second term by the UN General Assembly, unopposed and unanimously, on 21 June 2011, with effect from 1 January 2012. Mr. Ban describes his priorities as mobilising world leaders to deal with climate change, economic upheaval, pandemics and increasing pressures involving food, energy and water... Unfaithful Summary: The United Nations Secretary- General Ban Ki-moon was elected for a second term in 2007. Our Summary: The United Nations Secretary-General Ban Ki-moon was elected for a second term in 21 June 2011. Table 1: An example unfaithful summary. It suffers from extrinsic hallucination, where information not present in the source document was generated. Our method attempts to correct the unfaithful summary by replacing "2007" with an entity from the source with compatible semantic type (i.e. DATE). (Lin, 2004) or BERTSCORE (Zhang et al., 2020), current state of the art models (Liu and Lapata, 2019; Lewis …

Content truncated to 500 words