Improve Transformer Models with Better Relative Position Embeddings

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3327–3335 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3327 Improve Transformer Models with Better Relative Position Embeddings Zhiheng Huang AWS AI zhiheng@amazon.com Davis Liang AWS AI liadavis@amazon.com Peng Xu AWS AI pengx@amazon.com Bing Xiang AWS AI bxiang@amazon.com Abstract Transformer architectures rely on explicit po- sition encodings in order to preserve a notion of word order. In this paper, we argue that ex- isting work does not fully utilize position in- formation. For example, the initial proposal of a sinusoid embedding is fixed and not learn- able. In this paper, we first review absolute position embeddings and existing methods for relative position embeddings. We then pro- pose new techniques that encourage increased interaction between query, key and relative po- sition embeddings in the self-attention mecha- nism. Our most promising approach is a gen- eralization of the absolute position embedding, improving results on SQuAD1.1 compared to previous position embeddings approaches. In addition, we address the inductive property of whether a position embedding can be robust enough to handle long sequences. We demon- strate empirically that our relative position em- bedding method is reasonably generalized and robust from the inductive perspective. Finally, we show that our proposed method can be adopted as a near drop-in replacement for im- proving the accuracy of large models with a small computational budget. 1 Introduction The introduction of BERT (Devlin et al., 2018) has lead to new state-of-the-art results on various downstream tasks such as question answering and passage ranking. Variations of BERT, including RoBERTa (Liu et al., 2019b), XLNet (Yang et al., 2019), ALBERT (Lan et al., 2019) and T5 (Raffel et al., 2019) have been proposed. At its core, BERT is non-recurrent and based on self-attention; in or- der to model the dependency between elements at different positions in the sequence, BERT relies on position embeddings. With BERT, the input em- beddings are the sum of the token embeddings, seg- ment embeddings, and position embeddings. The position embedding encodes the absolute positions from 1 to maximum sequence length (usually 512). That is, each position has a learnable embedding vector. The absolute position embedding is used to model how a token at one position attends to another token at a different position. Recent work suggested removing the next sen- tence prediction (NSP) loss with training conducted solely on individual chunks of text (Liu et al., 2019a). In this setup, the notion of absolute po- sitions can be arbitrary depending on chunk start positions. Therefore, the association of a token to an absolute position is not well justified. Indeed, what really matters is the relative position or dis- tance between two tokens ti and tj, which is j − i. This phenomena has been realized and the relative position representation has been proposed in Shaw et al. (2018); Huang et al. (2018), in the context of encoder decoder machine translation and mu- sic generation respectively. Shaw et al. (2018) has been modified in transformer-XL …

Content truncated to 500 words