Modeling Past and Future for Neural Machine Translation

Modeling Past and Future for Neural Machine Translation Zaixiang Zheng∗ Nanjing University zhengzx@nlp.nju.edu.cn Hao Zhou∗ Toutiao AI Lab zhouhao.nlp@bytedance.com Shujian Huang Nanjing University huangsj@nlp.nju.edu.cn Lili Mou University of Waterloo doublepower.mou@gmail.com Xinyu Dai Nanjing University dxy@nlp.nju.edu.cn Jiajun Chen Nanjing University chenjj@nlp.nju.edu.cn Zhaopeng Tu Tencent AI Lab zptu@tencent.com Abstract Existing neural machine translation systems do not explicitly model what has been trans- lated and what has not during the decoding phase. To address this problem, we propose a novel mechanism that separates the source information into two parts: translated PAST contents and untranslated FUTURE contents, which are modeled by two additional recur- rent layers. The PAST and FUTURE contents are fed to both the attention model and the de- coder states, which provides Neural Machine Translation (NMT) systems with the knowl- edge of translated and untranslated contents. Experimental results show that the proposed approach significantly improves the perfor- mance in Chinese-English, German-English, and English-German translation tasks. Specif- ically, the proposed model outperforms the conventional coverage model in terms of both the translation quality and the alignment error rate.† 1 Introduction Neural machine translation (NMT) generally adopts an encoder-decoder framework (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014), where the encoder summarizes the source sentence into a source context vector, and the de- coder generates the target sentence word-by-word based on the given source. During translation, the decoder implicitly serves several functionalities at the same time: *Equal contributions. †Our code can be downloaded from https://github. com/zhengzx-nlp/past-and-future-nmt. 1. Building a language model over the target sen- tence for translation fluency (LM). 2. Acquiring the most relevant source-side in- formation to generate the current target word (PRESENT). 3. Maintaining what parts in the source have been translated (PAST) and what parts have not (FUTURE). However, it may be difficult for a single recur- rent neural network (RNN) decoder to accomplish these functionalities simultaneously. A recent suc- cessful extension of NMT models is the attention mechanism (Bahdanau et al., 2015; Luong et al., 2015), which makes a soft selection over source words and yields an attentive vector to represent the most relevant source parts for the current decoding state. In this sense, the attention mechanism sepa- rates the PRESENT functionality from the decoder RNN, achieving significant performance improve- ment. In addition to PRESENT, we address the impor- tance of modeling PAST and FUTURE contents in machine translation. The PAST contents indicate translated information, whereas the FUTURE con- tents indicate untranslated information, both being crucial to NMT models, especially to avoid under- translation and over-translation (Tu et al., 2016). Ideally, PAST grows and FUTURE declines during the translation process. However, it may be difficult for a single RNN to explicitly model the above pro- cesses. In this paper, we propose a novel neural machine translation system that explicitly models PAST and FUTURE contents with two additional RNN layers. The RNN modeling the PAST contents (called PAST layer) starts from scratch and accumulates the in- 145 Transactions of the Association for Computational Linguistics, vol. …

Content truncated to 500 words