Multi-View Source Ablation for Faithful Summarization

Findings of the Association for Computational Linguistics: EACL 2023, pages 2029–2047 May 2-6, 2023 ©2023 Association for Computational Linguistics Multi-View Source Ablation for Faithful Summarization Shuyang Cao1∗ Liang Ma2 Di Lu2 Robert L. Logan IV2 Joel Tetreault2 Alejandro Jaimes2 1University of Michigan, Ann Arbor 2Dataminr Inc. caoshuy@umich.edu {lma,dlu,rlogan,jtetreault,ajaimes}@dataminr.com Abstract In this paper, we present MUFASSA (Multi- view Faithfulness Scoring via Source Abla- tion), a metric for evaluating faithfulness of abstractive summaries, and for guiding train- ing of more faithful summarizers. For evalua- tion, MUFASSA employs different strategies (e.g., masking entity mentions) to first remove information from the source document to form multiple ablated views. Then, the faithfulness level of each token in a generated summary is measured by the difference between the to- ken generation probabilities when given the original document and the ablated document as inputs to trained summarizers. For training, MUFASSA uses a novel word truncation objec- tive that drops unfaithful tokens located by MU- FASSA in both the decoder input and output. Alignments with human-annotated faithfulness labels on AGGREFACT show that MUFASSA is comparable to or better than existing metrics built on classifiers or QA models pre-trained on other tasks. In experiments on summariza- tion with XSum and CNN/DailyMail, models trained with word truncation using MUFASSA outperform competitive methods according to both automatic faithfulness metrics and human assessments. 1 Introduction Automatic text summarization systems have made great strides with the use of large pre-trained mod- els, which enable more precise identification of salient content in the document and generation of summaries with human-level fluency (Lewis et al., 2020; Raffel et al., 2020). However, model- generated summaries frequently contain unfaith- ful information that either contradict the source text or cannot be verified (Kryscinski et al., 2020), creating risks in real-world deployment of auto- matic text summarization models and motivating the development of models targeting more faithful summaries (Cao et al., 2018). ∗ Work done during an internship at Dataminr. Summary Scottish Labour leader Kezia Dugdale has said she will make decisions about the party’s future in Scotland. Original Document … Dugdale was joined at The Meadows by shadow Scottish secretary Ian Murray, Labour's only MP in … Mask Entities and Proper Nouns … [MASK] was joined at [MASK] by shadow [MASK] secretary [MASK], [MASK] 's only [MASK] in … Shuffle Tokens … joined leader Labour's be took thatI loss Dugdale chance I Scottish deputy change any MSP said… Original: 0.168 ---------- Mask: 0.219 Shuffle: 0.008 Empty: 0.115 Original: 0.924 ---------- Mask: 0.949 Shuffle: 0.846 Empty: 0.730 make Kezia Output Probabilities Given Different Inputs Empty Document 0.160 ✅ -0.025 ❌ Figure 1: MUFASSA estimates the faithfulness level of each summary token as the difference between proba- bilities given by trained summarizers with the original source document and the ablated document chosen for the token (e.g., shuffling tokens for verbs). The large difference for “make” indicates it is faithful to the doc- ument, while the small difference for “Kezia” indicate an unfaithful token. As overlap-based metrics such as ROUGE (Lin, …

Content truncated to 500 words