Simple Local Attentions Remain Competitive for Long-Context Tasks

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1975 - 1986 July 10-15, 2022 ©2022 Association for Computational Linguistics Simple Local Attentions Remain Competitive for Long-Context Tasks Wenhan Xiong, Barlas O˘guz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Wen-tau Yih, Yashar Mehdad Meta AI {xwhan,barlaso,anchit,xilun,dianaml,omerlevy,scottyih,mehdad}@fb.com Abstract Many NLP tasks require processing long con- texts beyond the length limit of existing pre- trained models. To scale these models to longer text sequences, many efficient long- range attention variants have been recently pro- posed. Despite the abundance of research in this direction, it is difficult to gauge the rel- ative effectiveness of these models in practi- cal use cases, e.g., if we apply these models following the pretrain-and-finetune paradigm. In this work, we aim to conduct a thorough analysis of these emerging models with large- scale and controlled experiments. For each at- tention variant, we pretrain large-size models using the same long-doc corpus and then fine- tune these models for real-world long-context tasks. Our findings reveal pitfalls of a widely- used long-range benchmark and show that the other efficient attentions fail to outperform the simple local-window attention after stan- dard pretraining. Further analysis of local- attention variants suggests that even the com- monly used attention-window overlap is not necessary to achieve good downstream results — using disjoint local attentions, we are able to build a simpler and more efficient long-doc QA model that matches the performance of Longformer (Beltagy et al., 2020) with half of its pretraining compute.1 1 Introduction The quadratic complexity of Transformer archi- tectures makes it prohibitive to apply large state- of-the-art pretrained models to full-length docu- ments. To efficiently handle longer text while still maintaining the capacity of attention-based models, a long list of efficient attention variants have been proposed and many claim to effec- tively capture long-range dependencies. Typical paradigms of these architecture innovations involve 1The code to replicate our experiments can be found at https://github.com/pytorch/fairseq/ tree/main/examples/xformers learnable sparse attention patterns (Kitaev et al., 2020; Tay et al., 2020; Roy et al., 2021), fixed lo- cal patterns (Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020) and attention matrix approximation methods (Wang et al., 2020; Choro- manski et al., 2021; Xiong et al., 2021). While most of these studies have reported numbers on long sequence inputs, they tend to adopt quite dif- ferent benchmarks. For instance, Reformer (Ki- taev et al., 2020) is tested on the 64k-chunk en- wik8 dataset for unidirectional language model- ing; Performer (Choromanski et al., 2021) reports masked language modeling (MLM) perplexity on the PG-19 book corpus and protein sequences; Lin- former (Wang et al., 2020) reports MLP perplex- ity with various input length, while most of the documents in their pretrain corpus are short doc- uments.2 The divergence of evaluation protocols makes it hard to compare the relative performance of each attention variant and it is also unknown how they perform well in more practical use …

Content truncated to 500 words