The NLP Task Effectiveness of Long-Range Transformers

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3774–3790 May 2-6, 2023 ©2023 Association for Computational Linguistics The NLP Task Effectiveness of Long-Range Transformers Guanghui Qin Yukun Feng Benjamin Van Durme Department of Computer Science, Johns Hopkins University {gqin2, yfeng55, vandurme}@jhu.edu Abstract Transformer models cannot easily scale to long sequences due to their O(N 2) time and space complexity. This has led to Transformer vari- ants seeking to lower computational complex- ity, such as Longformer and Performer. While such models have theoretically greater effi- ciency, their effectiveness on real NLP tasks has not been well studied. We benchmark 7 variants of Transformer models on 5 difficult NLP tasks and 7 datasets. We design experi- ments to isolate the effect of pretraining and hy- perparameter settings, to focus on their capacity for long-range attention. Moreover, we present various methods to investigate attention behav- iors to illuminate model details beyond metric scores. We find that the modified attention in long-range transformers has advantages on con- tent selection and query-guided decoding, but they come with previously unrecognized draw- backs such as insufficient attention to distant tokens and accumulated approximation error. 1 Introduction Transformer-based models (Vaswani et al., 2017) have advanced the state of the art in natural lan- guage processing. However, their quadratic time and space complexity hinder their application on long texts. Various proposals have been made to ad- dress these concerns (Tay et al., 2020c), with math- ematical guarantees on improved time or space. These models have been evaluated primarily via perplexity (Dai et al., 2019) and non-NLP bench- marks (Tay et al., 2020b). These metrics may not be ideal (Sun et al., 2021) and may not reflect per- formance on complex NLP tasks (Arutiunian et al., 2020; Thorne et al., 2021). We argue these met- rics have not been sufficient for the development of efficient Transformers and their practical appli- cation on long texts, and that existing benchmarks are insufficient guides for architecture selection. It is not straightforward to have a fair and side- by-side comparison among those models due to the differences between their pretraining and hyperpa- rameter settings (Tay et al., 2020c), and the metrics alone cannot convey detailed information about the self-attention blocks (Sun et al., 2021). We wish to fairly validate the effectiveness of long-range attention techniques, and to uncover the underlying factors behind model behaviors. We critique the re- liance on perplexity evaluations in previous work, experimenting with five difficult, long-text NLP tasks. These tasks cover typical NLP modeling sce- narios: token or span-level prediction, sequence- level classification, and sequence-to-sequence gen- eration. To our knowledge, this is the first work to evaluate long-range transformers on such a wide spectrum of representative NLP tasks. To verify the key features of long-range trans- formers, we ablate distant attention to measure what they gain from long-range mechanisms. For models without pretrained checkpoints, we migrate parameters from their prototype models for fairness. We cover 3 main kinds of long-range transformers, including …

Content truncated to 500 words