Recurrent Positional Embedding for Neural Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1361–1367, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1361 Recurrent Positional Embedding for Neural Machine Translation Kehai Chen, Rui Wang∗, Masao Utiyama, and Eiichiro Sumita National Institute of Information and Communications Technology (NICT), Kyoto, Japan {khchen, wangrui, mutiyama, eiichiro.sumita}@nict.go.jp Abstract In the Transformer network architecture, positional embeddings are used to encode order dependencies into the input representa- tion. However, this input representation only involves static order dependencies based on discrete numerical information, that is, are independent of word content. To address this issue, this work proposes a recurrent positional embedding approach based on word vector. In this approach, these recurrent positional embeddings are learned by a recurrent neural network, encoding word content-based order dependencies into the input representation. They are then integrated into the existing multi-head self- attention model as independent heads or part of each head. The experimental results revealed that the proposed approach improved translation performance over that of the state- of-the-art Transformer baseline in WMT’14 English-to-German and NIST Chinese-to- English translation tasks. 1 Introduction Transformer translation systems (Vaswani et al., 2017), without recurrent and convolutional neural networks, rely on a positional embedding (PE) approach to encode order information into the input representation. PE is typically learned based on the position index of each word and is added to corresponding word embedding. This allows the Transformer to encode order dependencies between words in addition to the words themselves. Finally, the Transformer uses these combined vectors as the input to self- attention networks (SANs), achieving state-of-the- art translation performance with several language pairs (Vaswani et al., 2017; Dou et al., 2018; Zhang et al., 2018a; Marie et al., 2018, 2019). ∗Corresponding author In spite of their success, the input representation only involves static order dependencies based on discrete numerical information. That is, any word in the entire vocabulary has the same PE on the same position index. As a result, the dependencies encoded by the original PEs are independent of word content, which may further hinder the improvement of translation capacity. Recently, Chen et al. (2018) and Hao et al. (2019) introduced the additional source representation learned by an RNN-based encoder into Transformer to alleviate this issue, and reported improvements on the WMT’14 English-to-German translation task. Inspired by their works (Chen et al., 2018; Hao et al., 2019), we propose an simple and efficient recurrent positional embedding approach capture order dependencies based on word contentto in a sentence, thus learning a more effective sentence representation for the Transformer. In addition, we designed two simple multi-head self-attentions to introduce these learned RPEs and original input representation into the existing Transformer model for enhancing the sentence representation of Transformer. Experimental results on WMT’14 English-to-German and NIST Chinese-to-English translation tasks show that our models significantly improved translation performance over a strong Transformer baseline. 2 Background 2.1 Input Representation In the Transformer network architecture (Vaswani et …

Content truncated to 500 words