Enhancing Factual Consistency of Abstractive Summarization

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 718–733 June 6–11, 2021. ©2021 Association for Computational Linguistics 718 Enhancing Factual Consistency of Abstractive Summarization Chenguang Zhu1, William Hinthorn1, Ruochen Xu1, Qingkai Zeng2, Michael Zeng1, Xuedong Huang1, Meng Jiang2 1 Microsoft Cognitive Services Group 2 University of Notre Dame {chezhu,wihintho,ruox,nzeng,xdh}@microsoft.com {qzeng,mjiang2}@nd.edu Abstract Automatic abstractive summaries are found to often distort or fabricate facts in the article. This inconsistency between summary and orig- inal text has seriously impacted its applicabil- ity. We propose a fact-aware summarization model FASUM to extract and integrate factual relations into the summary generation process via graph attention. We then design a factual corrector model FC to automatically correct factual errors from summaries generated by ex- isting systems. Empirical results1 show that the fact-aware summarization can produce ab- stractive summaries with higher factual consis- tency compared with existing systems, and the correction model improves the factual consis- tency of given summaries via modifying only a few keywords. 1 Introduction Text summarization models aim to produce an abridged version of long text while preserving salient information. Abstractive summarization is a type of such models that can freely generate sum- maries, with no constraint on the words or phrases used. This format is closer to human-edited sum- maries and is both flexible and informative. Thus, there are numerous approaches to produce abstrac- tive summaries (See et al., 2017; Paulus et al., 2017; Dong et al., 2019; Gehrmann et al., 2018). However, one prominent issue with abstractive summarization is factual inconsistency. It refers to the hallucination phenomenon that the summary sometimes distorts or fabricates the facts in the article. Recent studies show that up to 30% of the summaries generated by abstractive models contain such factual inconsistencies (Kry´sci´nski et al., 2019b; Falke et al., 2019), raising concerns about the credibility and usability of these systems. 1We provide the prediction results of all models at https: //github.com/zcgzcgzcg1/FASum/. Article Real Madrid ace Gareth Bale treated himself to a Sunday evening BBQ... The Welsh wizard was ... scoring twice and assisting another in an impressive victory... Cristiano Ronaldo scored five goals against Granada on Sunday ... BOTTOMUP ... The Real Madrid ace scored five goals against Granada on Sunday. The Welsh wizard was in impressive form for... SEQ2SEQ ...Gareth Bale scored five and assisted another in an impressive win in Israel... FASUM (Ours) ...Gareth Bale scored twice and helped his side to a sensational 9-1 win. Cristiano Ronaldo scored five goals against Granada on Sunday... Table 1: Example article and summary excerpts from CNN/DailyMail dataset. Table 1 demonstrates an example article and ex- cerpts of generated summaries. As shown, the ar- ticle mentions that Real Madrid ace Gareth Bale scored twice and Cristiano Ronaldo scored five goals. However, both BOTTOMUP (Gehrmann et al., 2018) and SEQ2SEQ wrongly states that Bale scored five goals. Comparatively, our model FASUM generates a summary that correctly ex- hibits the fact in the article. And as shown …

Content truncated to 500 words