Learning with Rejection for Abstractive Text Summarization

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9768 - 9780 December 7-11, 2022 ©2022 Association for Computational Linguistics Learning with Rejection for Abstractive Text Summarization Meng Cao Yue Dong Jingyi He Jackie Chi Kit Cheung Mila / McGill University, Montréal, QC, Canada {meng.cao@mail, yue.dong2@mail, jingyi.he@mail, jcheung@cs}.mcgill.ca Abstract State-of-the-art abstractive summarization sys- tems frequently hallucinate content that is not supported by the source document, mainly due to noise in the training dataset. Existing meth- ods opt to drop the noisy samples or tokens from the training set entirely, reducing the ef- fective training set size and creating an artificial propensity to copy words from the source. In this work, we propose a training objective for abstractive summarization based on rejection learning, in which the model learns whether or not to reject potentially noisy tokens. We fur- ther propose a regularized decoding objective that penalizes non-factual candidate summaries during inference by using the rejection prob- ability learned during training. We show that our method considerably improves the factual- ity of generated summaries in automatic and human evaluations when compared to five base- line models, and that it does so while increasing the abstractiveness of the generated summaries. 1 1 Introduction Abstractive summarization has progressed signifi- cantly in recent years, as evidenced by automatic evaluation scores like ROUGE (Lin, 2004). How- ever, existing abstractive summarization systems are prone to produce non-factual summaries, which contain information that is not supported by the source (Kryscinski et al., 2019; Maynez et al., 2020; Durmus et al., 2020; Zhao et al., 2020; Kryscinski et al., 2020; Narayan et al., 2021). According to re- cent studies, one cause of this phenomenon is noise in the training set, such as unsupported facts in the references (Kang and Hashimoto, 2020; Goyal and Durrett, 2021; Raunak et al., 2021). Specifically, the widely-used negative log likelihood objective aims to match the underlying data distribution. If summarization models are trained on noisy datasets 1https://github.com/mcao516/rej-summ containing erroneous references, they will tend to imitate and even amplify undesired generations (Kang and Hashimoto, 2020). Different approaches have been proposed to re- duce non-factual hallucinations. Falke et al. (2019); Zhao et al. (2020) employ re-ranking based ap- proaches to up-rank candidate summaries based on their factuality scores from out-of-the-box fac- tual consistency evaluation systems. Kang and Hashimoto (2020); Goyal and Durrett (2021); Goyal et al. (2022) use loss truncation-based method that drops high loss sentences/tokens dur- ing training. Cao et al. (2022) propose reinforce- ment learning (RL)-based methods that penalize hallucinated tokens with negative rewards. One instance of the reference summary contain- ing unsupported information is shown in Table 1. We can see that the reference summary contains de- tails on the precise amount of government funding (£12.8m), yet this amount is absent from the source document. This kind of noise in the reference sum- mary is highly deceptive and could cause the sum- marization model to hallucinate. Furthermore, the noise can “break” the beam search process since the phrases associated …

Content truncated to 500 words