A Simple and Effective Positional Encoding for Transformers

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2974–2988 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2974 A Simple and Effective Positional Encoding for Transformers Pu-Chin Chen∗, Henry Tsai∗, Srinadh Bhojanapalli∗, Hyung Won Chung, Yin-Wen Chang, Chun-Sung Ferng Google Research Abstract Transformer models are permutation equivari- ant. To supply the order and type informa- tion of the input tokens, position and seg- ment embeddings are usually added to the in- put. Recent works proposed variations of po- sitional encodings with relative position en- codings achieving better performance. Our analysis shows that the gain actually comes from moving positional information to atten- tion layer from the input. Motivated by this, we introduce Decoupled posItional attEntion for Transformers (DIET), a simple yet effec- tive mechanism to encode position and seg- ment information into the Transformer mod- els. The proposed method has faster training and inference time, while achieving compet- itive performance on GLUE, XTREME and WMT benchmarks. We further generalize our method to long-range transformers and show performance gain. 1 Introduction Transformers are sequence-to-sequence models that achieve state of the art performance in many Natural Language Processing (NLP) tasks, such as machine translation, language modeling and ques- tion answering (Vaswani et al., 2017; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2020). Trans- formers have two major components: self-attention and a position-wise feed forward layer. Both are permutation equivariant and are not sensitive to the order of input tokens. To make these mod- els position-aware, the position information of the input words is typically added as an additional em- bedding to the input token embeddings (Vaswani et al., 2017). For example, input embedding (W ) of a sentence is added to the position embeddings (P ), resulting in input W + P to the Transformer. These position embeddings only depend on the lo- cation the word appears. For multi-segment tasks, ∗ The authors contribute equally to this paper. Corre- sponding author email: puchin@google.com additional segment embeddings can be added just like the position embeddings (Devlin et al., 2018). There have been multiple works exploring differ- ent ways to include position information in Trans- formers (Shaw et al., 2018; Yang et al., 2019; Raf- fel et al., 2020). Many of those note the advan- tages of using a relative position encoding scheme over absolute position encodings (see also Fig 1). However what causes this difference is not clear. Yun et al. (2020) have shown that Transformers with absolute position encodings are universal ap- proximators of all sequence to sequence functions, proving that absolute position encodings can cap- ture the position information. Hence what causes the superiority of relative position encodings? A systematic study and understanding of the benefits and drawbacks of different position encoding meth- ods is missing. Ke et al. (2020) hypothesised that the cross correlation between word and position embeddings while computing attention could be the cause of poor performance of absolute position encodings. However such cross terms are present in some …

Content truncated to 500 words