Explore Better Relative Position Embeddings from Encoding Perspective for Transformer Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2989–2997 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2989 Explore Better Relative Position Embeddings from Encoding Perspective for Transformer Models Anlin Qu1, Jianwei Niu1∗, Shasha Mo2∗ 1 State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing 100191, China 2 School of Cyber Science and Technology, Beihang University, Beijing 100191, China {anlin781205936, niujianwei, moshasha}@buaa.edu.cn Abstract Relative position embedding (RPE) is a suc- cessful method to explicitly and efficaciously encode position information into Transformer models. In this paper, we investigate the potential problems in Shaw-RPE and XL-RPE, which are the most representa- tive and prevalent RPEs, and propose two novel RPEs called Low-level Fine-grained High-level Coarse-grained (LFHC) RPE and Gaussian Cumulative Distribution Function (GCDF) RPE. LFHC-RPE is an improvement of Shaw-RPE, which enhances the perception ability at medium and long relative positions. GCDF-RPE utilizes the excellent properties of the Gaussian function to amend the prior en- coding mechanism in XL-RPE. Experimental results on nine authoritative datasets demon- strate the effectiveness of our methods empir- ically. Furthermore, GCDF-RPE achieves the best overall performance among five different RPEs. 1 Introduction Recently, the fully attention-based Transformer model (Vaswani et al., 2017) has achieved state- of-the-art results across a range of natural language processing (NLP) tasks, including reading compre- hension (Yu et al., 2018), machine translation (Raf- fel et al., 2020), natural language inference (Guo et al., 2019), unsupervised pretraining (Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019), etc. Since the self-attention blocks in vanilla Transformer are entirely invariant to sequence order, which is one of the most important features of natural language, how to explicitly encode position information is crucial for the current Transformer based models. The original method is to use absolute position embedding (APE), such as pre-defined sinusoidal functions (Vaswani et al., 2017) or fully data-driven learnable parameter embeddings (Devlin et al., ∗ Corresponding author. 2019; Radford et al., 2019), to integrate position in- formation into contextual representation. Although APE can significantly help the Transformer model learn the contextual representation of the tokens at different positions, Ke et al. (2020) pointed out that the coupled method in APE is unreasonable. Be- sides, APE itself also has many defects, such as the limitation of processing long sequences (Liu et al., 2020) and the gradual loss of position information (Al-Rfou et al., 2019). To address the drawbacks mentioned above of APE, Shaw et al. (2018); Dai et al. (2019) further proposed the relative position embedding (RPE), which incorporates carefully designed temporal bias term into the self-attention module to encode the relative distance between any two tokens. RPE has been proven to be more effective than APE, and thus it is adopted by many excellent pre-trained lan- guage models (Yang et al., 2019; Song et al., 2020; Dai et al., 2020). Despite the success of RPE, the existing methods are not perfect. Although Huang et al. (2020) has made …

Content truncated to 500 words