Long-Span Summarization via Local Attention and Content Selection

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6026–6041 August 1–6, 2021. ©2021 Association for Computational Linguistics 6026 Long-Span Summarization via Local Attention and Content Selection Potsawee Manakul and Mark J. F. Gales Department of Engineering, University of Cambridge pm574@cam.ac.uk, mjfg@eng.cam.ac.uk Abstract Transformer-based models have achieved state-of-the-art results in a wide range of nat- ural language processing (NLP) tasks includ- ing document summarization. Typically these systems are trained by fine-tuning a large pre- trained model to the target task. One is- sue with these transformer-based models is that they do not scale well in terms of mem- ory and compute requirements as the input length grows. Thus, for long document sum- marization, it can be challenging to train or fine-tune these models. In this work, we ex- ploit large pre-trained transformer-based mod- els and address long-span dependencies in ab- stractive summarization using two methods: local self-attention; and explicit content se- lection. These approaches are compared on a range of network configurations. Experi- ments are carried out on standard long-span summarization tasks, including Spotify Pod- cast, arXiv, and PubMed datasets. We demon- strate that by combining these methods, we can achieve state-of-the-art results on all three tasks in the ROUGE scores. Moreover, with- out a large-scale GPU card, our approach can achieve comparable or better results than exist- ing approaches.1 1 Introduction Transformer-based models (Vaswani et al., 2017) are ubiquitously state-of-art across many natural language processing (NLP) tasks, including sum- marization. To achieve the best results, the com- munity has trained ever larger transformer models on larger amount of data, and/or more task-specific optimization objectives (Devlin et al., 2019; Raf- fel et al., 2020; Lewis et al., 2020; Brown et al., 2020). In long document summarization, the input 1Our code is available at https://github.com/ potsawee/longsum0. sequences could be more than an order of mag- nitude longer than the limits of these transformer models. Although the limits can be extended, train- ing large transformer models on long sequences is expensive and may not be possible on a standard GPU card because of the self-attention mechanism that grows quadratically with sequence length. To tackle the quadratic characteristic, recent works have modified self-attention mechanism and proposed variants of the transformer such that the quadratic complexity is reduced (Tay et al., 2020b; Kitaev et al., 2020; Child et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020). However, pre-trained weights of the mod- ified models are not readily available. In contrast, standard models such as BERT (Devlin et al., 2019) or BART (Lewis et al., 2020) have been trained on various target tasks, including text summarization (Liu and Lapata, 2019b). This allows practition- ers to achieve good performance with less training time. Thus, we are interested in exploiting pre- trained models for long-span summarization tasks. We study a range of design configurations empir- ically and theoretically in regards to memory and compute requirements as well …

Content truncated to 500 words