Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9359–9368 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9359 Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems Potsawee Manakul and Mark J. F. Gales Department of Engineering, University of Cambridge pm574@cam.ac.uk, mjfg@eng.cam.ac.uk Abstract Transformer models have achieved state-of- the-art results in a wide range of NLP tasks including summarization. Training and in- ference using large transformer models can be computationally expensive. Previous work has focused on one important bottleneck, the quadratic self-attention mechanism in the en- coder. Modified encoder architectures such as LED or LoBART use local attention pat- terns to address this problem for summariza- tion. In contrast, this work focuses on the transformer’s encoder-decoder attention mech- anism. The cost of this attention becomes more significant in inference or training ap- proaches that require model-generated histo- ries. First, we examine the complexity of the encoder-decoder attention. We demon- strate empirically that there is a sparse sen- tence structure in document summarization that can be exploited by constraining the atten- tion mechanism to a subset of input sentences, whilst maintaining system performance. Sec- ond, we propose a modified architecture that selects the subset of sentences to constrain the encoder-decoder attention. Experiments are carried out on abstractive summarization tasks, including CNN/DailyMail, XSum, Spo- tify Podcast, and arXiv.1 1 Introduction The Transformer architecture (Vaswani et al., 2017) with large-scale pre-training has become the de- facto approach for a wide range of NLP tasks, from classification (Devlin et al., 2019) to seq2seq (Raf- fel et al., 2020). Training and inference using large transformer models can be computationally expen- sive because the self-attention’s time and memory grow quadratically with sequence length. Hence, there has been significant interest in efficient trans- former architectures. A number of approaches have 1Our code is available at https://github.com/ potsawee/encdec_attn_sparse. been proposed to tackle the quadratic complexity, and a comprehensive survey on efficient transform- ers has been compiled in Tay et al. (2020). Most existing approaches are developed for encoder-only architectures. For seq2seq tasks, efficient models such as BigBird (Zaheer et al., 2020) or LED (Belt- agy et al., 2020) consist of an efficient encoder with the vanilla decoder. For long-document summa- rization, this combination has been shown effective because the major bottleneck is the encoder self- attention (Manakul and Gales, 2021). The attention mechanisms in the decoder consist of self-attention and encoder-decoder attention. Techniques such as local attention are applicable to self-attention in both the encoder and decoder, while this work focuses on the encoder-decoder attention. When humans produce a summary, the infor- mation conveyed by each word/part in the sum- mary is likely drawn from some key sentences in the original document. Inspired by this, we hy- pothesize that if the encoder-decoder attention is constrained dynamically to salient sentences, the computation cost will be reduced. For instance, sentence-level structures for the encoder-decoder attention have been shown effective in the tra- ditional RNN encoder-decoder attention (Cohan et al., 2018; Li et al., 2019; …

Content truncated to 500 words