Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization

Findings of the Association for Computational Linguistics: EACL 2023, pages 2089–2105 May 2-6, 2023 ©2023 Association for Computational Linguistics Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization Markus Dreyer1 Mengwen Liu1 Feng Nan1 Sandeep Atluri1 Sujith Ravi2∗ Amazon1 SliceX2 AI {mddreyer, mengwliu, nanfen, satluri}@amazon.com ravi.sujith@gmail.com Abstract Neural models for abstractive summarization tend to generate output that is fluent and well- formed but lacks semantic faithfulness, or fac- tuality, with respect to the input documents. In this paper, we analyze the tradeoff between abstractiveness and factuality of generated sum- maries across multiple datasets and models, us- ing extensive human evaluations of factuality. In our analysis, we visualize the rates of change in factuality as we gradually increase abstrac- tiveness using a decoding constraint, and we observe that, while increased abstractiveness generally leads to a drop in factuality, the rate of factuality decay depends on factors such as the data that the system was trained on. We introduce two datasets with human factuality judgements; one containing 10.2k generated summaries with systematically varied degrees of abstractiveness; the other containing 4.2k summaries from five different summarization models. We propose new factuality metrics that adjust for the degree of abstractiveness, and we use them to compare the abstractiveness- adjusted factuality of previous summarization works, providing baselines for future work.1 1 Introduction Summarization is the task of generating a seman- tically faithful, well-formed and concise text rep- resentation of the input. Automatically generated summaries have traditionally been extractive (Luhn, 1958; Edmundson, 1969; Neto et al., 2002; Erkan and Radev, 2004; Wong et al., 2008), leading to issues with readability and coherence, as different extracted fragments may not fit well when taken out of their original contexts (Poibeau and Sag- gion, 2012). Researchers have also invested in methods for abstractive summarization, aiming to paraphrase the input documents’ main points ∗Work conducted during his position at Amazon. 1Code and data are available at https: //github.com/amazon-science/ abstractive-factual-tradeoff. Figure 1: Three successively more abstractive sum- maries generated from the same input article, with MINT abstractiveness scores (Section 2.1) of 46.1%, 67.2%, 79.5%. Fragments extracted from the input are marked from red (longer fragments) to yellow (shorter frag- ments). The bottom summary has factual errors. without borrowing their exact lexical expressions (Radev and McKeown, 1998; Saggion and La- palme, 2002; Ganesan et al., 2010; Genest and Lapalme, 2012; Radford et al., 2019; Gehrmann et al., 2019; Lewis et al., 2019; Zhang et al., 2020). Abstractive summaries generated by today’s neural models tend to be fluent and well-formed, but lack semantic faithfulness (Cao et al., 2017; Kryscinski et al., 2019). Observed rates of factual errors in abstractive summaries have ranged from 30% to over 75% (Cao et al., 2017; Maynez et al., 2020). The research community is developing automatic factuality metrics (Wang et al., 2020; Kryscinski et al., 2020; Goodrich et al., 2019; Goyal and Dur- rett, 2020; Ribeiro et al., 2022) and methods that attempt to increase factuality (Fan et al., 2018; Scialom et al., 2019; Zhang et al., …

Content truncated to 500 words