Factual Error Correction for Abstractive Summarization Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6251–6258, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6251 Factual Error Correction for Abstractive Summarization Models Meng Cao Yue Dong Jiapeng Wu Jackie Chi Kit Cheung School of Computer Science, McGill University, Montreal, QC, Canada MILA, Montreal, QC, Canada {meng.cao@mail, yue.dong2@mail, jiapeng.wu@mail, jcheung@cs}.mcgill.ca Abstract Neural abstractive summarization systems have achieved promising progress, thanks to the availability of large-scale datasets and models pre-trained with self-supervised meth- ods. However, ensuring the factual consis- tency of the generated summaries for abstrac- tive summarization systems is a challenge. We propose a post-editing corrector module to ad- dress this issue by identifying and correcting factual errors in generated summaries. The neural corrector model is pre-trained on arti- ficial examples that are created by applying a series of heuristic transformations on refer- ence summaries. These transformations are in- spired by an error analysis of state-of-the-art summarization model outputs. Experimental results show that our model is able to correct factual errors in summaries generated by other neural summarization models and outperforms previous models on factual consistency eval- uation on the CNN/DailyMail dataset. We also find that transferring from artificial error correction to downstream settings is still very challenging1. 1 Introduction Self-supervised methods have achieved success in a wide range of NLP tasks, and automatic summarization is no exception (Liu and Lapata, 2019; Lewis et al., 2019; Zhang et al., 2019a; Shi et al., 2019; Fabbri et al., 2019). These state-of- the-art abstractive summarization models typically finetune pre-trained transformer-based models on a summarization dataset (Vaswani et al., 2017). Despite significant improvements over previous methods in terms of automatic evaluation scores such as ROUGE (Lin, 2004), ensuring factual con- sistency of the generated summary with respect to the source remains challenging. For example, Cao 1Our data and code is available at https://github. com/mcao610/Factual-Error-Correction Source: Jerusalem (CNN)The flame of remembrance burns in Jerusalem, and a song of memory haunts Valerie Braham as it never has before. This year, Israel’s Memorial Day commemoration is for bereaved family members such as Braham. “Now I truly understand everyone who has lost a loved one,” Braham said. (...) Original: France’s memorial day commemoration is for bereaved family members as braham. (inconsistent) After Correction: Israel’s memorial day commemoration is for bereaved family members as braham. (consistent) Table 1: An example of an inconsistent system- generated summary and the output summary from our correction model. In this case, “France” is successfully corrected as “Israel”. et al. (2018) claims that about 30% of summaries generated by abstractive models contain factual er- rors, which greatly limits their practicality. Different approaches have been proposed to de- tect or ensure the factual consistency of generated summaries, including using fact extraction or ap- plying attention on fact triples (Cao et al., 2018; Zhang et al., 2019b; Goodrich et al., 2019), apply- ing natural language inference or question answer- ing models for consistency checking (Falke et al., 2019; Li et al., 2018; Wang et al., 2020) and train- …

Content truncated to 500 words