Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1410 - 1421 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization Faisal Ladhak1∗ Esin Durmus2∗ He He3 Claire Cardie4 Kathleen McKeown1 1Columbia University 3New York University2Stanford University 4Cornell University Abstract Despite recent progress in abstractive summa- rization, systems still suffer from faithfulness errors. While prior work has proposed models that improve faithfulness, it is unclear whether the improvement comes from an increased level of extractiveness of the model outputs as one naive way to improve faithfulness is to make summarization models more extrac- tive. In this work, we present a framework for evaluating the effective faithfulness of summa- rization systems, by generating a faithfulness- abstractiveness trade-off curve that serves as a control at different operating points on the ab- stractiveness spectrum. We then show that the baseline system as well as recently proposed methods for improving faithfulness, fail to con- sistently improve over the control at the same level of abstractiveness. Finally, we learn a selector to identify the most faithful and ab- stractive summary for a given document, and show that this system can attain higher faith- fulness scores in human evaluations while be- ing more abstractive than the baseline system on two datasets. Moreover, we show that our system is able to achieve a better faithfulness- abstractiveness trade-off than the control at the same level of abstractiveness. 1 Introduction Generating abstractive summaries of documents has been a long-standing goal of summarization. While there has been tremendous progress towards this goal (Kry´sci´nski et al., 2018; Dong et al., 2019; Zhang et al., 2019; Lewis et al., 2020), abstractive summarization systems still suffer from faithful- ness errors (Cao et al., 2018), generating informa- tion that is not present in the original text. This has led to an increased research in faithfulness evalua- tion of summarization systems (Falke et al., 2019; Kryscinski et al., 2020; Durmus et al., 2020) as ∗Equal contribution. Corresponding author for queries: faisal@cs.columbia.edu. well as methods to improve faithfulness of gen- erated summaries (Kang and Hashimoto, 2020; Chen et al., 2021). Intuitively, one straightfor- ward way of improving faithfulness of generated summaries is to copy a larger amount of content from the source article (i.e. more extraction). Thus, any methods that increase the level of extractive- ness, whether intentionally or not, would improve faithfulness. Without reported extractiveness, it is unclear whether prior improvements mainly arise from increased extractiveness. We argue that in order to make progress in abstractive summariza- tion, it is important to tease apart faithfulness im- provements due to increased extractiveness versus improvements due to improved abstraction. In order to tease this apart, we develop a frame- work for evaluating progress in faithfulness, by con- sidering the effective faithfulness, i.e. the improve- ment in faithfulness over a baseline system (con- trol) operating at the same level of extractiveness. In particular, we split the training examples into dif- …

Content truncated to 500 words