Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13522–13537 July 9-14, 2023 ©2023 Association for Computational Linguistics Dissecting Transformer Length Extrapolation via The Lens of Receptive Field Analysis Ta-Chung Chi Carnegie Mellon University tachungc@andrew.cmu.edu Ting-Han Fan Princeton University tinghanf@princeton.edu Alexander I. Rudnicky Carnegie Mellon University air@cs.cmu.edu Peter J. Ramadge Princeton University ramadge@princeton.edu Abstract Length extrapolation permits training a trans- former language model on short sequences that preserves perplexities when tested on sub- stantially longer sequences. A relative posi- tional embedding design, ALiBi, has had the widest usage to date. We dissect ALiBi via the lens of receptive field analysis empow- ered by a novel cumulative normalized gradi- ent tool. The concept of receptive field further allows us to modify the vanilla Sinusoidal po- sitional embedding to create Sandwich, the first parameter-free relative positional embed- ding design that truly length information uses longer than the training sequence. Sandwich shares with KERPLE and T5 the same logarith- mic decaying temporal bias pattern with learn- able relative positional embeddings; these elu- cidate future extrapolatable positional embed- ding design. 1 Introduction The length of input sequences is an important hy- perparameter choice for pretraining a transformer language model. A vanilla transformer language model has a quadratic training cost w.r.t Ltr, the training sequence length. As the value of Ltr in- creases, cost becomes impractical. However, we can use the model for substantially longer evalu- ation sequence lengths Lex ≫ Ltr as gradients no longer need to be recorded. The discrepancy between Ltr and Lex motivates the task of length extrapolation (Press et al., 2022): Can a trans- former language model maintain equally good, if not better, perplexities when longer sequences are used in the testing stage? Several extrapolatable transformer language models have been proposed including ALiBi (Press et al., 2022) and KERPLE (Chi et al., 2022), of which the relative positional embedding design is hypothesized to be critical to success. Empirically, they extrapolate to Lex ≫ Ltr much better than other absolute and relative positional embeddings − 1 2' × *")" *#)" *#)# *$)" *$)# *$)$ 0 1 0 2 1 0 3 0 1 2 *%)" *%)# *%)$ *%)% Figure 1: ALiBi. For a transformer language model with H attention heads, the range of h is n · 8 H , where n = {1 . . . H}. Left = self-attention matrix, right = temporal biases matrix. − *")" *#)" *#)# *$)" *$)# *$)$ 0 0 0 ∞ 0 0 ∞ 0 0 ∞ *%)" *%)# *%)$ *%)% Figure 2: Windowed Attention. This is the same de- sign as Longformer (Beltagy et al., 2020). We limit the context window size to w = 2 in this example. Left = self-attention matrix, right = temporal biases matrix. including Sinusoidal (Vaswani et al., 2017), Ro- tary (Su et al., 2021), and T5 (Raffel et al., 2020), resulting in the adoption of ALiBi for the recently released Bloom (Scao et al., 2022) model. Despite the significant …

Content truncated to 500 words