X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7100 - 7110 December 7-11, 2022 ©2022 Association for Computational Linguistics X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization Subhajit Chaudhury, Sarathkrishna Swaminathan, Chulaka Gunasekara, Maxwell Crouse, Srinivas Ravishankar, Daiki Kimura, Keerthiram Murugesan, Ramón Fernandez Astudillo, Tahira Naseem, Pavan Kapanipathi, Alexander Gray {subhajit, sarath.swaminathan, chulaka.gunasekara, maxwell.crouse}@ibm.com {srini, keerthiram.murugesan, ramon.astudillo, alexander.gray}@ibm.com daiki@jp.ibm.com, tnaseem@us.ibm.com, kapanipa@us.ibm.com IBM Research Abstract Abstractive summarization models often pro- duce factually inconsistent summaries that are not supported by the original article. Recently, a number of fact-consistent evaluation tech- niques have been proposed to address this issue; however, a detailed analysis of how these met- rics agree with one another has yet to be con- ducted. In this paper, we present X-FACTOR, a cross-evaluation of three high-performing fact-aware abstractive summarization methods. First, we show that summarization models are often fine-tuned on datasets that contain factu- ally inconsistent summaries and propose a fact- aware filtering mechanism that improves the quality of training data and, consequently, the factuality of these models. Second, we propose a corrector module that can be used to improve the factual consistency of generated summaries. Third, we present a re-ranking technique that samples summary instances from the output distribution of a summarization model and re- ranks the sampled instances based on their fac- tuality. Finally, we provide a detailed cross- metric agreement analysis that shows how tun- ing a model to output summaries based on a particular factuality metric influences factuality as determined by the other metrics. Our goal in this work is to facilitate research that improves the factuality and faithfulness of abstractive summarization models. 1 Introduction In this work, we consider the task of automatic text summarization, i.e., the task of generating a concise summary of a given document that preserves its most salient information (Maybury, 1999). This is a prominent task that has been applied across a va- riety of contexts and domains, such as the summa- rization of legal documents (Kanapala et al., 2019), automatic news summarization (Fabbri et al., 2019), and summarization of dialog between agents and clients (Feigenblat et al., 2021). Article: Kremlin spokesman Dmitry Peskov said on Thurs- day that the Russian air force would continue its support of the Syrian armed forces. He also urged Washington to deliver on a pledge to separate moderate Syrian opposition fighters from "terrorists" .. .The Russian foreign ministry said a US refusal to co-operate would be a gift to "terrorists". The US and Russia have been negotiating for months to try to secure a cessation of hostilities but the latest truce collapsed last week after only a few days and attacks on eastern Aleppo have since intensified . . . BART: Russia has said it will continue to support the Syrian government despite a partial truce collapsing. Rerank (FactGraph): Russia has said it will continue its air strikes in Syria despite a partial truce collapsing. Figure 1: Factual hallucinations on an XSum article by BART (Lewis et al., …

Content truncated to 500 words