SHAPE: Shifted Absolute Position Embedding for Transformers

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3309–3321 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3309 SHAPE: Shifted Absolute Position Embedding for Transformers Shun Kiyono♠,♥ Sosuke Kobayashi♥,♦ Jun Suzuki♥,♠ Kentaro Inui♥,♠ ♠ RIKEN ♥ Tohoku University ♦ Preferred Networks, Inc. shun.kiyono@riken.jp, sosk@preferred.jp, {jun.suzuki, inui}@tohoku.ac.jp Abstract Position representation is crucial for building position-aware representations in Transformers. Existing position representations suffer from a lack of generalization to test data with un- seen lengths or high computational cost. We investigate shifted absolute position embedding (SHAPE) to address both issues. The basic idea of SHAPE is to achieve shift invariance, which is a key property of recent successful position representations, by randomly shifting absolute positions during training. We demonstrate that SHAPE is empirically comparable to its coun- terpart while being simpler and faster1. 1 Introduction Position representation plays a critical role in self- attention-based encoder-decoder models (Trans- formers) (Vaswani et al., 2017), enabling the self-attention to recognize the order of input se- quences. Position representations have two cat- egories (Dufter et al., 2021): absolute position embedding (APE) (Gehring et al., 2017; Vaswani et al., 2017) and relative position embedding (RPE) (Shaw et al., 2018). With APE, each po- sition is represented by a unique embedding, which is added to inputs. RPE represents the position based on the relative distance between two tokens in the self-attention mechanism. RPE outperforms APE on sequence-to-sequence tasks (Narang et al., 2021; Neishi and Yoshinaga, 2019) due to extrapolation, i.e., the ability to gen- eralize to sequences that are longer than those observed during training (Newman et al., 2020). Wang et al. (2021) reported that one of the key properties contributing to RPE’s superior perfor- mance is shift invariance2, the property of a func- tion to not change its output even if its input is shifted. However, unlike APE, RPE’s formulation 1The code is available at https://github.com/ butsugiri/shape. 2Shift invariance is also known as translation invariance. Multi-Head Attention Add & Norm Feed Forward Add & Norm Input Embedding + John yelled at Kevin Position 0 1 2 3 4 k-1 k k+1 k+2 k+3 Absolute Position (a)Embedding (APE) Shifted APE (SHAPE) (c) Transformer Relative Position Embedding (RPE) (b) Relative Distance John yelled at Kevin !!"# {%&',)*+,&} !."# {%&',)*+,&} -2 -1 0 1 2 Key Value Shifted by random offset k Figure 1: Overview of position representations. (a) APE and (c) SHAPE consider absolute positions in the input layer, whereas (b) RPE considers the relative position of a given token pair in the self-attention mechanism. strongly depends on the self-attention mechanism. This motivated us to explore a way to incorporate the benefit of shift invariance in APE. A promising approach to achieving shift invari- ance while using absolute positions is to randomly shift positions during training. A similar idea can be seen in several contexts, e.g., computer vision (Goodfellow et al., 2016) and question-answering in NLP (Geva et al., 2020). APE is no exception; a random shift should force Transformer to cap- ture the …

Content truncated to 500 words