Gradient-Based Adversarial Factual Consistency Evaluation for Abstractive Summarization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4102–4108 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4102 Gradient-based Adversarial Factual Consistency Evaluation for Abstractive Summarization Zhiyuan Zeng1∗, Jiaze Chen2, Weiran Xu1, Lei Li3∗ 1Beijing University of Posts and Telecommunications, Beijing, China {zengzhiyuan,xuweiran}@bupt.edu.cn 2ByteDance AI Lab 3University of California, Santa Barbara chenjiaze@bytedance.com lilei@cs.ucsb.edu Abstract Neural abstractive summarization systems have gained significant progress in recent years. However, abstractive summarization of- ten produce inconsisitent statements or false facts. How to automatically generate highly abstract yet factually correct summaries? In this paper, we proposed an efficient weak- supervised adversarial data augmentation ap- proach to form the factual consistency dataset. Based on the artificial dataset, we train an evaluation model that can not only make ac- curate and robust factual consistency discrim- ination but is also capable of making inter- pretable factual errors tracing by backpropa- gated gradient distribution on token embed- dings. Experiments and analysis conduct on public annotated summarization and fac- tual consistency datasets demonstrate our ap- proach effective and reasonable. Our codes can be found at https://github.com/ parZival27/GrAdualCC 1 Introduction Text summarization aims to produce a simplified version of the source document while retaining salient information. Abstractive summarization is a branch of methods in which generation text is free from constraint on the tokens that appeared in the source. These methods are extensively studied since its flexibility and generalization ability(See et al., 2017; Paulus et al., 2018; Gehrmann et al., 2018; Dong et al., 2019a). However, a challenge in abstractive summarization is the trade-off between abstractiveness and factual consistency. Recent studies show that about 30% of the summaries gen- erated by abstractive models contain facts errors toward source documents. The proportion will rise further as the data abstractiveness increases(Cao et al., 2018; Durmus et al., 2020; Kryscinski et al., 2020), causing factual checking an essential pro- cess to verify the credibility and usability of models. ∗Work is done while at ByteDance. article: The Swift Archway Cranford 545 caravan was stolen from a site in Yaxley, Cambridgeshire, on Thursday night. Davis tweeted "My touring caravan was stolen.. even though it was locked up with hitch & wheel lock!". (...) Davis has played the role of Professor Flitwick in the Harry Potter films and Nikabrik in The Chronicles of Narnia: Prince Caspian. (...) claim: A caravan locked by Harry has been stolen from a site in cambridgeshire. reference: A caravan locked by Davis has been stolen from a site in cambridgeshire. Table 1: An incorrect summary generated by XSUM. In Table 1, we propose an inconsistent generation example, where the blue part support factual con- sistency and the red part leads to factual errors. Previous approaches for detecting or boosting factual consistency can be divided into three kinds. (1) Employ information extraction tools to extract facts and leveraging it by building additional ob- jective (Cao et al., 2018; Goodrich et al., 2019; Zhang et al., 2020a; Zhu et al., 2021). (2) Use natural language inference or question answering …

Content truncated to 500 words