Globalizing BERT-based Transformer Architectures for Long Document Summarization

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1792–1810 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1792 Globalizing BERT-based Transformer Architectures for Long Document Summarization Quentin Grail NAVER LABS Europe, Meylan, France Univ. Grenoble Alpes, CNRS, LIG, Grenoble, France quentin.grail@naverlabs.com Julien Perez NAVER LABS Europe, Meylan, France julien.perez@naverlabs.com Eric Gaussier Univ. Grenoble Alpes, CNRS, LIG, Grenoble, France eric.gaussier@imag.fr Abstract Fine-tuning a large language model on down- stream tasks has become a commonly adopted process in the Natural Language Processing (NLP) (Wang et al., 2019). However, such a process, when associated with the current transformer-based (Vaswani et al., 2017) archi- tectures, shows several limitations when the target task requires to reason with long doc- uments. In this work, we introduce a novel hierarchical propagation layer that spreads in- formation between multiple transformer win- dows. We adopt a hierarchical approach where the input is divided in multiple blocks indepen- dently processed by the scaled dot-attentions and combined between the successive lay- ers. We validate the effectiveness of our ap- proach on three extractive summarization cor- pora of long scientific papers and news articles. We compare our approach to standard and pre-trained language-model-based summariz- ers and report state-of-the-art results for long document summarization and comparable re- sults for smaller document summarization. 1 Introduction Language model pre-training has become a key component to improve performances on a major- ity of Natural Language Processing (NLP) tasks (Wang et al., 2019). Most of the recent competi- tive architectures (Devlin et al., 2019; Lan et al., 2020; Liu et al., 2019b; Radford et al., 2018) are based on the efficient transformer layer introduced in Vaswani et al. (2017). BERT (Devlin et al., 2019) is one of these architectures that has been widely adopted for comprehension and generation tasks. It is a multi-layer transformer network, pre- trained with different self-supervised objectives. Numerous variations of transformer architectures have been proposed to improve this approach (Lan et al., 2020; Liu et al., 2019b; Radford et al., 2018). However, this type of process is only evaluated on tasks composed of relatively short input text, GLUE (Wang et al., 2019), SQUAD (Rajpurkar et al., 2016), SWAG (Zellers et al., 2018). Indeed, for the tasks that require reasoning with longer doc- uments, this approach exhibits several limitations. The transformer self-attention memory quadrati- cally increases with the number of input tokens, making it technically impossible to compute on document-scale sequences. In addition, they usu- ally require to define a fixed maximum input length, typically of 512 tokens, at the pre-training stage. One solution is to pre-train the entire model on longer sequences. However, this will still require a massive computation power and will only push the length limitation further. Other alternatives have been proposed to extend multi-layer transformers architectures to longer sequences without modify- ing this maximum length limitation. The first one is to limit the input sequence to its first tokens by removing the text beyond the length limit. Obvi- ously, it …

Content truncated to 500 words