Leveraging Locality in Abstractive Text Summarization

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6081 - 6093 December 7-11, 2022 ©2022 Association for Computational Linguistics Leveraging Locality in Abstractive Text Summarization Yixin Liu1, Ansong Ni1, Linyong Nan1, Budhaditya Deb2, Chenguang Zhu2, Ahmed H. Awadallah2, Dragomir Radev1 1Yale University, 2Microsoft Research {yixin.liu, ansong.ni, linyong.nan, dragomir.radev}@yale.edu {Budha.Deb, chezhu, hassanam}@microsoft.com Abstract Neural attention models have achieved signifi- cant improvements on many natural language processing tasks. However, the quadratic mem- ory complexity of the self-attention module with respect to the input length hinders their applications in long text summarization. In- stead of designing more efficient attention mod- ules, we approach this problem by investigat- ing if models with a restricted context can have competitive performance compared with the memory-efficient attention models that main- tain a global context by treating the input as a single sequence. Our model is applied to in- dividual pages, which contain parts of inputs grouped by the principle of locality, during both the encoding and decoding stages. We em- pirically investigated three kinds of locality in text summarization at different levels of gran- ularity, ranging from sentences to documents. Our experimental results show that our model has a better performance compared with strong baseline models with efficient attention mod- ules, and our analysis provides further insights into our locality-aware modeling strategy.1 1 Introduction Neural abstractive summarization (Rush et al., 2015; Nallapati et al., 2016) is mainly formulated as a sequence-to-sequence (Sutskever et al., 2014) (Seq2Seq) problem. Neural attention models, e.g., Transformers (Vaswani et al., 2017), have been widely used for such Seq2Seq tasks, allowing effec- tive modeling of various dependencies in the input and output sequences. However, the self-attention module in such models introduces a quadratic mem- ory growth with respect to the input sequence length. Consequently, for long-text summarization datasets,2 recent works (Beltagy et al., 2020; Ki- taev et al., 2020; Zaheer et al., 2020) have explored 1We have made our code, results, and trained models pub- licly available at https://github.com/yixinL7/PageSum. 2For example, the average input document length in the arXiv dataset (Cohan et al., 2018) is more than 8,000 tokens. 0 200 400 600 800 1000 Sentence Distance 0.325 0.350 0.375 0.400 0.425 0.450 0.475 0.500 Average Similarity Figure 1: Intrinsic spatial locality in the arXiv dataset. The X-axis represents the distance of two sentences in source documents measured by the difference of their locations (indexes). Y-axis represents the average se- mantic similarity calculated by the cosine similarity between sentence embeddings, which are generated by a pre-trained sentence embedding model (Gao et al., 2021). The dash line shows the average similarity. using efficient attention to reduce the memory foot- print while still maintaining the same global con- text of a full-attention model – every input token can receive information from all the other input tokens. However, efficient attention is just an ap- proximation of full attention and can show lower performance compared with its counterpart (Kitaev et al., 2020). To investigate an alternative memory- efficient modeling approach, we argue …

Content truncated to 500 words