Improving Factual Consistency in Summarization with Compression-Based Post-Editing

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9149 - 9156 December 7-11, 2022 ©2022 Association for Computational Linguistics Improving Factual Consistency in Summarization with Compression-Based Post-Editing Alexander R. Fabbri Prafulla Kumar Choubey Jesse Vig Chien-Sheng Wu Caiming Xiong Salesforce AI Research {afabbri, pchoubey, jvig, wu.jason, cxiong}@salesforce.com Abstract State-of-the-art summarization models still struggle to be factually consistent with the in- put text. A model-agnostic way to address this problem is post-editing the generated sum- maries. However, existing approaches typically fail to remove entity errors if a suitable input entity replacement is not available or may insert erroneous content. In our work, we focus on re- moving extrinsic entity errors, or entities not in the source, to improve consistency while retain- ing the summary’s essential information and form. We propose to use sentence-compression data to train the post-editing model to take a summary with extrinsic entity errors marked with special tokens and output a compressed, well-formed summary with those errors re- moved. We show that this model improves factual consistency while maintaining ROUGE, improving entity precision by up to 30% on XSum, and that this model can be applied on top of another post-editor, improving entity precision by up to a total of 38%. We per- form an extensive comparison of post-editing approaches that demonstrate trade-offs between factual consistency, informativeness, and gram- maticality, and we analyze settings where post- editors show the largest improvements. 1 Introduction Text summarization aims to compress a long docu- ment(s) into a short and fluent form that preserves salient information. State-of-the-art models, how- ever, are often not factually consistent with the input they are conditioned on (Maynez et al., 2020; Fabbri et al., 2022). While recent modeling tech- niques have been proposed to improve factual con- sistency (Nan et al., 2021; Kang and Hashimoto, 2020; Aralikatte et al., 2021), a model-agnostic approach is to post-edit the summaries. Prior work in post-editing for factual consistency has focused on swapping inconsistent entities with those in the input (Dong et al., 2020; Lee et al., 2022), including by reranking the entity-replaced Source: The oil price collapse sent global markets reeling throughout 2015. ... Brent crude oil was up 3% at $37.60 per barrel for the day but down 35% over the year. System Summary Original Wall Street markets closed lower on the last trad- ing day of 2015 as oil prices languished at $28 a barrel for much of the year. Entity Swap No Change Rewrite Wall Street markets closed lower on the last trad- ing day of 2015 as oil prices languished at $37.60 per barrel ... but remain down 35% over the year. Delete Wall Street markets closed lower on 2015 as oil prices languished at a for much of the year. Compress (ours) Wall Street markets closed lower as oil prices languished for much of the year. Table 1: Example of post-editing approaches. Entity swap (Chen et al., 2021) does not modify the errors (red) if a suitable entity replacement is …

Content truncated to 500 words