Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9818 - 9830 December 7-11, 2022 ©2022 Association for Computational Linguistics Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling Vidhisha Balachandran♣ Hannaneh Hajishirzi♢♡ William W. Cohen♠ Yulia Tsvetkov♡ ♣Language Technologies Institute, Carnegie Mellon University ♢Allen Institute for Artificial Intelligence ♡Paul G. Allen School of Computer Science & Engineering, University of Washington ♠Google Research vbalacha@cs.cmu.edu, hannaneh@cs.washington.edu, wcohen@google.com, yuliats@cs.washington.edu Abstract Abstractive summarization models often gen- erate inconsistent summaries containing fac- tual errors or hallucinated content. Recent works focus on correcting factual errors in gen- erated summaries via post-editing. Such correc- tion models are trained using adversarial non- factual summaries constructed using heuristic rules for injecting errors. However, generat- ing non-factual summaries using heuristics of- ten does not generalize well to actual model errors. In this work, we propose to generate hard, representative synthetic examples of non- factual summaries through infilling language models. With this data, we train a more ro- bust fact-correction model to post-edit the sum- maries to improve factual consistency. Through quantitative and qualitative experiments on two popular summarization datasets— CNN/DM and XSum—we show that our approach vastly outperforms prior methods in correcting erro- neous summaries. Our model—FACTEDIT— improves factuality scores by over ∼11 points on CNN/DM and over ∼31 points on XSum on average across multiple summarization models, producing more factual summaries while main- taining competitive summarization quality.1 1 Introduction While modern summarization models generate highly fluent summaries that appear realistic (Lewis et al., 2020; Zhang et al., 2020), these models are prone to generating non-factual and sometimes en- tirely fabricated content (Cao et al., 2018; Goodrich et al., 2019; Maynez et al., 2020). With the in- creasing adoption of language generation tools in user-facing products, such unreliability poses se- vere risks, including the spread of misinformation, panic and other potentially harmful effects (Ranade et al., 2021; Hutson et al., 2021). Since it is difficult to control for factuality at training or inference time (Huang et al., 2021; 1Code and data available at https://github.com/ vidhishanair/FactEdit. The first vaccine for Ebola was approved by the FDA in 2019 in the US, five years after the initial outbreak in 2014. To produce the vaccine, scientists had to sequence the DNA of Ebola, then identify possible vaccines, and finally show successful clinical trials. Scientists say a vaccine for COVID-19 is unlikely to be ready this year, although clinical trials have already started. Scientists believe a vaccine for Covid-19 might not be ready this year. The first vaccine for Ebola took 5 years to be approved by the FDA. Scientists believe a vaccine for Ebola might not be ready this year. The first vaccine for Ebola took 5 years to be produced by the CBP. Incorrect Entity Incorrect Predicate Hallucination Source Generated Summary Corrected Summary Error Correction Figure 1: Model generated summaries often produce content which is factually inconsistent w.r.t. to the source. FACTEDIT rewrites these summaries by main- taining the abstractiveness but correcting factual …

Content truncated to 500 words