Factorizing Content and Budget Decisions in Abstractive Summarization of Long Documents

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6341 - 6364 December 7-11, 2022 ©2022 Association for Computational Linguistics Factorizing Content and Budget Decisions in Abstractive Summarization of Long Documents Marcio Fonseca Yftah Ziser Shay B. Cohen Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB m.fonseca@ed.ac.uk, yftah.ziser@ed.ac.uk, scohen@inf.ed.ac.uk Abstract We argue that disentangling content selection from the budget used to cover salient content improves the performance and applicability of abstractive summarizers. Our method, FAC- TORSUM1, does this disentanglement by factor- izing summarization into two steps through an energy function: (1) generation of abstractive summary views covering salient information in subsets of the input document (document views) ; (2) combination of these views into a final summary, following a budget and content guidance. This guidance may come from differ- ent sources, including from an advisor model such as BART or BigBird, or in oracle mode – from the reference. This factorization achieves significantly higher ROUGE scores on multiple benchmarks for long document summarization, namely PubMed, arXiv, and GovReport. No- tably, our model is effective for domain adapta- tion. When trained only on PubMed, it achieves a 46.29 ROUGE-1 score on arXiv, outperform- ing PEGASUS trained in domain by a large margin. Our experimental results indicate that the performance gains are due to more flexi- ble budget adaptation and processing of shorter contexts provided by partial document views. 1 Introduction Casting summarization as a language transduction problem is convenient given the existence of pow- erful neural sequence-to-sequence models that pro- duce high-quality textual outputs (Zhang et al., 2020; Lewis et al., 2020). However, this frame- work conflates multiple steps of the summarization decision-making process into a single feedforward step without taking into account the contextual fac- tors involved (Jones et al., 1999). One such decision depending on context factors is the quantity of information to be included in a summary, reflecting on the generated outputs’ 1Code is available at https://github.com/ thefonseca/factorsum. length. This factor is particularly relevant for long documents that cover many different aspects of interest for which different summaries may be suit- able. For instance, in samples from summarization datasets such as PubMed and arXiv (Cohan et al., 2018), there are many abstracts including terse pas- sages about the background or methods of the re- search. In contrast, others will add more details about those aspects. Often, those choices are due to the author’s preferences and do not necessarily represent an ideal summary for the document. Furthermore, current evaluation protocols based on n-gram overlap are sensitive to summary lengths (Sun et al., 2019). Generating summaries that match ground-truth lengths increases performance (see Section 4.1). Thus, recent progress in summa- rization may be the effect of better length prediction and not the actual summarization desideratum: a re- ductive transformation of the source text that keeps the important information (Jones et al., 1999). To address this issue, we propose to avoid bud- get information as a confounding …

Content truncated to 500 words