Efficient Attentions for Long Document Summarization

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419–1436 June 6–11, 2021. ©2021 Association for Computational Linguistics 1419 Efficient Attentions for Long Document Summarization Luyang Huang 1 Shuyang Cao1 Nikolaus Parulian2 Heng Ji2 Lu Wang1 1Computer Science and Engineering, University of Michigan, Ann Arbor, MI 2Department of Computer Science, University of Illinois at Urbana-Champaign, IL 1{lyhuang, caoshuy, wangluxy}@umich.edu 2{nnp2, hengji}@illinois.edu Abstract The quadratic computational and memory complexities of large Transformers have lim- ited their scalability for long document sum- marization. In this paper, we propose HEPOS, a novel efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source. We further conduct a systematic study of ex- isting efficient self-attentions. Combined with HEPOS, we are able to process ten times more tokens than existing models that use full atten- tions. For evaluation, we present a new dataset, GOVREPORT, with significantly longer docu- ments and summaries. Results show that our models produce significantly higher ROUGE scores than competitive comparisons, includ- ing new state-of-the-art results on PubMed. Human evaluation also shows that our mod- els generate more informative summaries with fewer unfaithful errors. 1 Introduction Long documents, such as scientific papers and gov- ernment reports, often discuss substantial issues at length, and thus are time-consuming to read, let alone to comprehend. Generating abstractive sum- maries can help readers quickly grasp the main topics, yet prior work has mostly focused on short texts (containing hundreds of words), e.g., news articles (Gehrmann et al., 2018; Liu and Lapata, 2019; Zhang et al., 2019). Model training efficiency and summary quality present a pair of challenges for long document summarization. State-of-the-art systems (Lewis et al., 2020; Zhang et al., 2019) are built upon Transformer (Vaswani et al., 2017), which uses at- tentions to compute pairwise relations between to- kens. Such framework has quadratic time and mem- ory complexities, and is too costly for long docu- ments 1. Solutions have been proposed to reduce 1For instance, to fine-tune BART on documents of 10K the calculation of encoder self-attentions (Wang et al., 2020c; Zaheer et al., 2020) by selectively at- tending to neighboring tokens (Beltagy et al., 2020; Child et al., 2019) or relevant words (Kitaev et al., 2020; Tay et al., 2020a). Yet, these methods do not apply to encoder-decoder attentions in summariza- tion models since they collaborate and dynamically pinpoint salient content in the source as the sum- mary is decoded. Truncation is commonly used to circumvent the issue. However, training on cur- tailed content further aggravates “hallucination” in existing abstractive models (Maynez et al., 2020). We argue that summarizing long documents (e.g., with thousands of words or more) requires ef- ficient handling of both types of attentions. To this end, we propose an efficient encoder-decoder atten- tion with head-wise positional strides (HEPOS), where the attention heads follow a strided pattern and have varying starting positions. HEPOS re- duces computational and memory costs while (1) maintaining the power of emphasizing …

Content truncated to 500 words