Improving Factual Consistency of Abstractive Summarization via Question Answering

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6881–6894 August 1–6, 2021. ©2021 Association for Computational Linguistics 6881 Improving Factual Consistency of Abstractive Summarization via Question Answering Feng Nan1 Cicero Nogueira dos Santos1 Henghui Zhu1 Patrick Ng1 Kathleen McKeown1,2 Ramesh Nallapati1 Dejiao Zhang1 Zhiguo Wang1 Andrew O. Arnold 1 Bing Xiang1 Amazon Web Services1, Columbia University2 {nanfen, cicnog, henghui, patricng, mckeownk rnallapa, dejiaoz, zhiguow, anarnld, bxiang}@amazon.com Abstract A commonly observed problem with the state- of-the art abstractive summarization models is that the generated summaries can be factually inconsistent with the input documents. The fact that automatic summarization may produce plausible-sounding yet inaccurate summaries is a major concern that limits its wide application. In this paper we present an approach to address factual consistency in summarization. We first propose an efficient automatic evaluation metric to measure factual consistency; next, we propose a novel learning algorithm that maximizes the proposed metric during model training. Through extensive experiments, we confirm that our method is effective in improving factual consistency and even overall quality of the summaries, as judged by both automatic metrics and human evaluation. 1 Introduction Recent advances in neural text generation have led to significant improvement in the quality of abstractive summarization (Radford et al., 2019; Gehrmann et al., 2019; Lewis et al., 2019). Despite this progress, there are still many limitations facing neural text summarization (Kryscinski et al., 2019), the most serious of which is the tendency to generate summaries that are not factually consistent with the input document; a factually consistent summary only contains statements that can be inferred from the source document. Recent studies show that about 30% of the summaries generated by neural network sequence-to-sequence (seq2seq) models suffer from fact fabrication (Cao et al., 2018). The standard training approach for seq2seq learning has been maximizing the log likelihood of the target given the input sequences (MLE). It has empirically performed well as a surrogate loss Input: ...“Klitschko doesn’t have the legs, the power that he used to,” said Lewis. “He has a chink in his armour after getting beat by Tyson Fury. Anthony Joshua is now taking that challenge, going after the man.” ... MLE: Anthony Joshua has a “chink in his armour” ahead of his world heavyweight title bout with Wladimir Klitschko, says former champion Lennox Lewis. CONSEQ: Wladimir Klitschko has a “chink in his armour” and is no match for British champion Anthony Joshua, says former world heavyweight champion Lennox Lewis. Table 1: Example summaries from the BART-large finetuned models on test set. Standard MLE training generates a factually inconsistent summary whereas our proposed CONSEQ is consistent. for evaluation metrics such as BLEU and ROUGE. This empirical success can be ascribed to the fact that both BLEU and ROUGE are directly linked to the n-gram overlap between the output and the target sequences, which can be efficiently learned via MLE. In contrast, metrics to capture factual consistency are much more …

Content truncated to 500 words