FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1010 - 1028 July 10-15, 2022 ©2022 Association for Computational Linguistics FACTPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization David Wan and Mohit Bansal University of North Carolina at Chapel Hill {davidwan,mbansal}@cs.unc.edu Abstract We present FACTPEGASUS, an abstractive summarization model that addresses the prob- lem of factuality during pre-training and fine- tuning: (1) We augment the sentence selec- tion strategy of PEGASUS’s (Zhang et al., 2020) pre-training objective to create pseudo- summaries that are both important and factual; (2) We introduce three complementary compo- nents for fine-tuning. The corrector removes hallucinations present in the reference sum- mary, the contrastor uses contrastive learning to better differentiate nonfactual summaries from factual ones, and the connector bridges the gap between the pre-training and fine- tuning for better transfer of knowledge. Exper- iments on three downstream tasks demonstrate that FACTPEGASUS substantially improves factuality evaluated by multiple automatic met- rics and humans. Our thorough analysis sug- gests that FACTPEGASUS is more factual than using the original pre-training objective in zero-shot and few-shot settings, retains factual behavior more robustly than strong baselines, and does not rely entirely on becoming more extractive to improve factuality.1 1 Introduction Abstractive summarization aims at generating short summaries that capture the essentials of a long document. Research in this challenging task has made significant progress with the help of large pre-trained models (Lewis et al., 2020; Raffel et al., 2020; Zhang et al., 2020). However, current mod- els suffer from the crucial problem of hallucina- tions (Maynez et al., 2020), where a summary con- tains facts or entities not present in the original document. Such unfaithful generation raises the question of whether the models can be trustwor- thy and used safely for real-world applications. To tackle this problem, many approaches propose post- processing models (Chen et al., 2021; Dong et al., 1Our code and data are publicly available at: https: //github.com/meetdavidwan/factpegasus. 2020; Liu and Liu, 2021), but such methods are often constrained by external resources to train ad- ditional correction or selection models. An alter- native line of works focuses on learning factuality directly during fine-tuning by filtering nonfactual training data (Goyal and Durrett, 2021; Nan et al., 2021) or, most recently, incorporating contrastive learning (Cao and Wang, 2021) to encourage gen- erating faithful summaries. In this work, we propose FACTPEGASUS, a model that addresses the problem of hallucina- tions for abstractive summarization holistically, by incorporating factuality into the whole train- ing pipeline: We tackle the lack of factuality ob- jective in pre-training and the presence of hallu- cinations in the downstream dataset during fine- tuning. Current pre-training objectives focus on improving the quality of the generated output in the downstream tasks but often overlook the factuality aspect. Thus, we explore incorporating factual- ity into the pre-training objective of PEGASUS (Zhang et al., 2020) (a state-of-the-art abstractive summarization model). The original objective, gap sentence generation (GSG), transforms any text into a …

Content truncated to 500 words