Analysis of Positional Encodings for Neural Machine Translation

Analysis of Positional Encodings for Neural Machine Translation Jan Rosendahl, Viet Anh Khoa Tran Weiyue Wang and Hermann Ney Human Language Technology and Pattern Recognition Group RWTH Aachen University, Germany {rosendahl,vtran,wwang}@i6.informatik.rwth-aachen.de Abstract In this work we analyze and compare the behavior of the Transformer architecture when using different positional en- coding methods. While absolute and relative positional en- coding perform equally strong overall, we show that rela- tive positional encoding is vastly superior (4.4% to 11.9% BLEU) when translating a sentence that is longer than any observed training sentence. We further propose and analyze variations of relative positional encoding and observe that the number of trainable parameters can be reduced without a performance loss, by using fixed encoding vectors or by removing some of the positional encoding vectors. 1. Introduction In this work we analyze the performances of different meth- ods to encode positional information in the Transformer ar- chitecture [1]. The Transformer architecture relies on self- attention layers to handle sequences of varying length. How- ever, self-attention layers themselves provide little positional information. Keys and values in self-attention are treated as sets without an ordering and reordering the queries simply results in a reordered output. While the decoder may get some positional information from the left-to-right masking scheme, the encoder does not have access to any positional information. To solve this problem, Transformer models rely on explicit positional information, typically in form of a (ab- solute) positional encoding vector that is added to each em- bedded word. Recent works suggest to use relative positional encodings instead. This can be done by modifying the self-attention layer, which considers each pair of tokens from the input sen- tence, to include a new vector that encodes the distances of the tokens [2, 3]. In this work we empirically analyze different positional encoding schemes for the use of machine translation. We compare absolute and relative positional encodings with a strong focus on their behaviour for different sentence lengths. Furthermore we propose and analyze different variations and combinations of absolute and relative positional encoding. 2. Related Work The Transformer [1] model consists of stacked encoder and decoder layers. Since the model uses self-attention layers to handle sequences, information about the sequence order has to be included explicitly into the model. Vaswani et al. [1] introduced the Transformer using absolute positional encod- ings based on sinusoid functions. Absolute positional encod- ings are also used in convolutional systems [4]. The posi- tional encodings are added to the input embeddings at the bottom of the encoder and decoder stacks. Shaw et al. [2] propose an alternative approach, in which the self-attention mechanism is extended to efficiently ac- count for representations of relative positions or distances be- tween tokens. The relative positional information is included by adding vectors, which represent the pairwise relationship between the current position and other positions, to the pro- jected keys and values. Our experiments are mainly based on this technique. Chen et al. [5] use bidirectional long short-term mem- ory (LSTM) [6] cells …

Content truncated to 500 words