Neural Machine Translation with Reordering Embeddings

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1787–1799 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1787 Neural Machine Translation with Reordering Embeddings Kehai Chen, Rui Wang∗, Masao Utiyama, and Eiichiro Sumita National Institute of Information and Communications Technology (NICT), Kyoto, Japan {khchen, wangrui, mutiyama, eiichiro.sumita}@nict.go.jp Abstract The reordering model plays an important role in phrase-based statistical machine translation. However, there are few works that exploit the reordering information in neural machine translation. In this paper, we propose a reordering mechanism to learn the reordering embedding of a word based on its contextual information. These reordering embeddings are stacked together with self-attention networks to learn sentence representation for machine translation. The reordering mechanism can be easily integrated into both the encoder and the decoder in the Transformer translation system. Experimental results on WMT’14 English-to- German, NIST Chinese-to-English, and WAT ASPEC Japanese-to-English translation tasks demonstrate that the proposed methods can significantly improve the performance of the Transformer translation system. 1 Introduction The reordering model plays an important role in phrase-based statistical machine translation (PB- SMT), especially for translation between distant language pairs with large differences in word order, such as Chinese-to-English and Japanese- to-English translations (Galley and Manning, 2008; Goto et al., 2013). Typically, the traditional PBSMT learns large-scale reordering rules from parallel bilingual sentence pairs in advance to form a reordering model. This reordering model is then integrated into the translation decoding process to ensure a reasonable order of translations of the source words (Chiang, 2005; Xiong et al., 2006; Galley and Manning, 2008). In contrast to the explicit reordering model for PBSMT, the RNN-based NMT (Sutskever et al., 2014; Bahdanau et al., 2015) depends on neural networks to implicitly encode order dependencies ∗Corresponding author between words in a sentence to generate a fluent translation. Inspired by a distortion method originating in SMT (Brown et al., 1993; Koehn et al., 2003; Al-Onaizan and Papineni, 2006), there is a quite recent preliminary exploration work for NMT (Zhang et al., 2017). They distorted the existing content-based attention by an additional position-based attention inside the fixed-size window, and reported a considerable improvement on the classical RNN-based NMT. This means that the word reordering information is also beneficial to the NMT. The Transformer (Vaswani et al., 2017) trans- lation system relies on self-attention networks (SANs), and has attracted growing interesting in the machine translation community. The Transformer generates an ordered sequence of positional embeddings by a positional encoding mechanism (Gehring et al., 2017a) to explicitly encode the order of dependencies between words in a sentence. The Transformer is adept at parallelizing of performing (multi-head) and stacking (multi-layer) SANs to learn the sentence representation to predict translation, and has delivered state-of-the-art performance on various translation tasks (Bojar et al., 2018; Marie et al., 2018). However, these positional embeddings focus on sequentially encoding order relations between words, and does not explicitly consider reordering information in a sentence, which may degrade the performance of Transformer intranslation …

Content truncated to 500 words