Shortformer: Better Language Modeling using Shorter Inputs

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5493–5505 August 1–6, 2021. ©2021 Association for Computational Linguistics 5493 Shortformer: Better Language Modeling Using Shorter Inputs Ofir Press1,2 Noah A. Smith1,3 Mike Lewis2 1Paul G. Allen School of Computer Science & Engineering, University of Washington 2Facebook AI Research 3Allen Institute for AI ofirp@cs.washington.edu Abstract Increasing the input length has been a driver of progress in language modeling with trans- formers. We identify conditions where shorter inputs are not harmful, and achieve perplex- ity and efficiency improvements through two new methods that decrease input length. First, we show that initially training a model on short subsequences before moving on to longer ones both reduces overall training time and, surprisingly, substantially improves perplex- ity. Second, we show how to improve the ef- ficiency of recurrence methods in transform- ers, which let models condition on previously processed tokens when generating sequences that exceed the maximal length the transformer can handle at once. Existing methods re- quire computationally expensive relative posi- tion embeddings; we introduce a simple alter- native of adding absolute position embeddings to queries and keys instead of to word embed- dings, which efficiently produces superior re- sults. We show that these recurrent models also benefit from short input lengths. Com- bining these techniques speeds up training by a factor of 1.65, reduces memory usage, and substantially improves perplexity on WikiText- 103, without adding any parameters.1 1 Introduction Scaling up transformer (Vaswani et al., 2017) lan- guage models (Radford et al., 2019; Lewis et al., 2019; Raffel et al., 2019; Brown et al., 2020) has been an important driver of progress in NLP. Lan- guage models require data to be segmented into subsequences for both training and inference: mem- ory constraints limit a language model to handling at most a few thousand tokens at once, while many training and evaluation datasets are much longer. 1Our code is available at https://github.com/ ofirpress/shortformer Recent work focuses on increasing the length of in- put subsequences, which determines the maximum number of tokens a model can attend to (Baevski and Auli, 2018; Sukhbaatar et al., 2019; Kitaev et al., 2020; Roy et al., 2020). We challenge the assumption that longer input subsequences are always better by showing that existing transformers do not always effectively use them. We then introduce new methods based on shorter input subsequences that improve runtime, memory efficiency, and perplexity. We first investigate how input subsequence length affects transformer language models (§3). Na¨ıve evaluation—where we split a large evalua- tion set into multiple nonoverlapping subsequences, each evaluated independently—initially supports the commonly-held belief that models that train and do inference on longer subsequences achieve better perplexity (Table 1, col. 3). However, when we evaluate each model with a sliding window (Baevski and Auli, 2018), out- putting one token at a time using the maximal amount of context, we find—surprisingly—that models using subsequences exceeding 1,024 to- kens do not further improve …

Content truncated to 500 words