DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1687 - 1698 May 22-27, 2022 c⃝2022 Association for Computational Linguistics DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization Ziming Mao∗ 1 Chen Henry Wu∗ 2 Ansong Ni1 Yusen Zhang3 Rui Zhang3 Tao Yu4 Budhaditya Deb5 Chenguang Zhu5 Ahmed H. Awadallah5 Dragomir Radev1 1 Yale University 2 Carnegie Mellon University 3 Penn State University 4 The University of Hong Kong 5 Microsoft Research ziming.mao@yale.edu, henrychenwu@cmu.edu Abstract Transformer-based models have achieved state-of-the-art performance on short-input summarization. However, they still struggle with summarizing longer text. In this pa- per, we present DYLE, a novel dynamic latent extraction approach for abstractive long-input summarization. DYLE jointly trains an extrac- tor and a generator and treats the extracted text snippets as the latent variable, allowing dynamic snippet-level attention weights dur- ing decoding. To provide adequate supervi- sion, we propose simple yet effective heuris- tics for oracle extraction as well as a con- sistency loss term, which encourages the ex- tractor to approximate the averaged dynamic weights predicted by the generator. We eval- uate our method on different long-document and long-dialogue summarization tasks: Gov- Report, QMSum, and arXiv. Experiment re- sults show that DYLE outperforms all exist- ing methods on GovReport and QMSum, with gains up to 6.1 ROUGE, while yielding strong results on arXiv. Further analysis shows that the proposed dynamic weights provide inter- pretability of our generation process.1 1 Introduction Transformer-based (Vaswani et al., 2017) pre- trained language models (PLMs) such as BART (Lewis et al., 2020a) and T5 (Raffel et al., 2020), have achieved state-of-the-art performance on short text summarization. However, due to the high mem- ory complexity of the full self-attention (Tay et al., 2020a), PLMs still struggle to handle long inputs (Rohde et al., 2021). Model efficiency and sum- mary quality present a pair of challenges (Huang et al., 2021): models need to capture information scattered across the long input while maintaining a low computational cost. ∗Equal Contributions. 1Our code is available at: https://github.com/ Yale-LILY/DYLE Generator Extractor Oracle loss Consistency loss Generation loss Document Query Dynamic weights Figure 1: Overview of our approach. The input is a document X (each x ∈ X is a sentence) and an op- tional query q, and the output is a summary y. Prior models tackled long input summarization mostly in four ways. First, sparse attention (Child et al., 2019; Beltagy et al., 2020; Tay et al., 2020b) is used to reduce the memory complexity of the Transformers so that they can attend to more to- kens. Second, extract-then-generate methods ex- tract salient texts from the input and then sum- marize the extracted texts. Extractors are either independently trained with full supervision (Zhong et al., 2021b) or optimized using reinforcement learning (Williams, 1992; Chen and Bansal, 2018; Bae et al., 2019; Bražinskas et al., 2021). Third, models are proposed to divide source text into sec- tions (Gidiotis and Tsoumakas, 2020; Wu et al., 2021; Liu et al., 2021) …

Content truncated to 500 words