CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633–6649 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6633 CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization Shuyang Cao and Lu Wang Computer Science and Engineering University of Michigan Ann Arbor, MI {caoshuy, wangluxy}@umich.edu Abstract We study generating abstractive summaries that are faithful and factually consistent with the given articles. A novel contrastive learn- ing formulation is presented, which leverages both reference summaries, as positive training data, and automatically generated erroneous summaries, as negative training data, to train summarization systems that are better at dis- tinguishing between them. We further de- sign four types of strategies for creating neg- ative samples, to resemble errors made com- monly by two state-of-the-art models, BART and PEGASUS, found in our new human an- notations of summary errors. Experiments on XSum and CNN/Daily Mail show that our con- trastive learning framework is robust across datasets and models. It consistently produces more factual summaries than strong compar- isons with post error correction, entailment- based reranking, and unlikelihood training, ac- cording to QA-based factuality evaluation. Hu- man judges echo the observation and find that our model summaries correct more errors. 1 Introduction Large pre-trained Transformers have yielded re- markable performance on abstractive summariza- tion (Liu and Lapata, 2019; Lewis et al., 2020; Zhang et al., 2020a) with impeccable fluency, yet their summaries often contain factually inconsis- tent content (Maynez et al., 2020; Zhang et al., 2020b; Goyal and Durrett, 2020), even for state- of-the-art models. Three types of remedies have been proposed: running a separately learned error correction component (Dong et al., 2020), remov- ing noisy training samples (Nan et al., 2021; Goyal and Durrett, 2021), and modifying the Transformer architecture (Huang et al., 2020; Zhu et al., 2021). Yet they either rely on heuristically created data for error handling, falling short of generalization, or require learning a large number of new parameters, and summary informativeness is often sacrificed. XSum Article: The Fermanagh MLA Phil Flanagan tweeted after Tom Elliott appeared on a BBC radio programme in May 2014. . . . “I wonder if he will reveal how many people he harassed and shot as a member of the UDR.”. . . Contrastive learning (our method): A Sinn Féin MLA has been ordered to apologise and pay compensation to a former member of the Ulster Defence Regiment (UDR). Cross-entropy: A Sinn Féin MLA has agreed to pay com- pensation to a former Ulster Unionist Party (UDR) MP after he tweeted that he had harassed and shot people as a member of the party. Entailment reranking: A Sinn Féin MLA has agreed to pay compensation to a former Ulster Unionist Party (UDR) councillor for a tweet he sent about him. Unlikelihood: An MLA has been ordered to apologise and pay compensation to a former loyalist MP for a remark he made about him while serving in the Ministry of Defence. Figure 1: Sample article and system summaries by dif- …

Content truncated to 500 words