What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6840–6849, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6840 What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding Yu-An Wang Yun-Nung Chen Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan r08922019@csie.ntu.edu.tw y.v.chen@ieee.org Abstract In recent years, pre-trained Transformers have dominated the majority of NLP benchmark tasks. Many variants of pre-trained Trans- formers have kept breaking out, and most fo- cus on designing different pre-training objec- tives or variants of self-attention. Embedding the position information in the self-attention mechanism is also an indispensable factor in Transformers however is often discussed at will. Therefore, this paper carries out an empirical study on position embeddings of mainstream pre-trained Transformers, which mainly focuses on two questions: 1) Do po- sition embeddings really learn the meaning of positions? 2) How do these different learned position embeddings affect Transformers for NLP tasks? This paper focuses on providing a new insight of pre-trained position embed- dings through feature-level analysis and empir- ical experiments on most of iconic NLP tasks. It is believed that our experimental results can guide the future work to choose the suitable positional encoding function for specific tasks given the application property.1 1 Introduction Word ordering often determines the meaning of a sentence; therefore how to utilize the position information of a word sequence has been an impor- tant topic in NLP and widely investigated recently. A common approach for modeling word ordering is to use recurrent neural networks (RNN), such as long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) or gated recurrent unit (GRU) (Chung et al., 2014), which use a hidden state to represent the information of an ordered se- quence and update model weights by backpropaga- tion through time (BPTT) (Werbos, 1990); thus the 1The source code is available at: https://github. com/MiuLab/PE-Study ordering information can be modeled by this struc- ture. However, RNN and BPTT are very inefficient in modern GPU computation due to the difficulty of parallelization with the time dependency. To solve this problem, recent work, such as convolutional seq2seq (Gehring et al., 2017) and Transformers (Vaswani et al., 2017) which apply convolutional neural network (CNN) (LeCun et al., 1995) and self-attention respectively, succeed to eliminate the time dependency to take the computational advan- tage of GPU. Instead of storing the information of ordered sequences, these models utilize the posi- tion information by using a feature-level positional encoding. For example, convolutional seq2seq pro- posed learnable position embeddings to represent the positions in a sequence. Recently, various pre-trained Transformer lan- guage models keep breaking state-of-the-art results in numerous NLP tasks. There are many different ways to pre-train a Transformer language model. For example, using an encoder, decoder, or the whole part of the Transformer, adapting the self- attention masks, or training with different objec- tives (Devlin et al., 2018; Liu et al., 2019; Radford et al., 2018, 2019; Lewis et al., 2019; Raffel et al., 2019; Yang …

Content truncated to 500 words