Dynamic Position Encoding for Transformers

Proceedings of the 29th International Conference on Computational Linguistics, pages 5076–5084 October 12–17, 2022. 5076 Dynamic Position Encoding for Transformers Joyce Zheng∗ Huawei Noah’s Ark Lab jy6zheng@uwaterloo.ca Mehdi Rezagholizadeh Huawei Noah’s Ark Lab mehdi.rezagholizadeh@huawei.com Peyman Passban † Amazon passban.peyman@gmail.com Abstract Recurrent models have been dominating the field of neural machine translation (NMT) for the past few years. Transformers (Vaswani et al., 2017) have radically changed it by proposing a novel architecture that relies on a feed-forward backbone and self-attention mechanism. Although Transformers are pow- erful, they could fail to properly encode se- quential/positional information due to their non- recurrent nature. To solve this problem, posi- tion embeddings are defined exclusively for each time step to enrich word information. However, such embeddings are fixed after train- ing regardless of the task and word ordering system of the source and target languages. In this paper, we address this shortcoming by proposing a novel architecture with new po- sition embeddings that take the order of the target words into consideration. Instead of us- ing predefined position embeddings, our solu- tion generates new embeddings to refine each word’s position information. Since we do not dictate the position of the source tokens and we learn them in an end-to-end fashion, we refer to our method as dynamic position encoding (DPE). We evaluated the impact of our model on multiple datasets to translate from English to German, French, and Italian and observed meaningful improvements in comparison to the original Transformer. 1 Introduction In statistical machine translation (SMT), the gen- eral task of translation consists of reducing the input sentence into smaller units (also known as statistical phrases), selecting an optimal translation for each unit, and placing them in the correct order (Koehn, 2009). The last step, which is also referred to as the reordering problem, is a great source of complexity and importance, which is handled with ∗ Work done while Joyce Zheng was an intern at Huawei † Work done while Peyman Passban was at Huawei. a variety of statistical as well as string- and tree- based solutions (Bisazza and Federico, 2016). As evident in SMT, the structure and position of words within a sentence is crucial for accurate translation. The importance of such information can also be explored in NMT and for Transform- ers. Previous literature, such as Chen et al. (2020), demonstrated that source input sentences enriched with target order information have the capacity to improve translation quality in neural models. They showed that position encoding seems to play a key role in translation and this motivated us to further explore this area. Since Transformers have a non-recurrent archi- tecture, they could face problems when encoding sequential data. As a result, they require an explicit summation of the input embeddings with position encoding to provide information about the order of each word. However, this approach falsely assumes that the correct position of each word is always its original position in the source sentence. This inter- pretation might be true when only considering the …

Content truncated to 500 words