An Analysis of Source Context Dependency in Neural Machine Translation

An Analysis of Source Context Dependency in Neural Machine Translation Xutai Ma Electrical and Computer Engineering Johns Hopkins University xutai ma@jhu.edu Ke Li Electrical and Computer Engineering Johns Hopkins University kli26@jhu.edu Philipp Koehn Computer Science Department Johns Hopkins University phi@jhu.edu Abstract The encoder-decoder with attention model has become the state of the art for machine translation. However, more investigations are still needed to understand the inter- nal mechanism of this end-to-end model. In this paper, we focus on how neural machine translation (NMT) models con- sider source information while decoding. We propose a numerical measurement of source context dependency in the NMT models and analyze the behaviors of the NMT decoder with this measurement un- der several circumstances. Experimental results show that this measurement is an appropriate estimate for source context de- pendency and consistent over different do- mains. 1 Introduction Neural machine translation (NMT) with encoder- decoder structure and attention mechanism (Bah- danau et al., 2015; Luong et al., 2015) has achieved great success on several machine translation tasks. Different from phrase based systems, neural ma- chine translation is trained end-to-end and learns the alignment and translation jointly. At each decoding step, the alignment is pre- dicted in the attention layer and represented as a distribution over words in a source sequence. Then the source context information, which is an attention-weighted sum over encoder hidden states, is fed into the decoder for the prediction of the next word. c⃝ 2018 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. The decoder in a NMT model is similar to a re- current neural network language model (RNNLM) (Mikolov et al., 2010), with additional input from the source side. It takes the previous hidden state, the previous predicted target word embedding, and source context information as inputs and produces a distribution over the next target words. This end-to-end approach can achieve state-of- the-art performance on several machine translation tasks. Joint training of the translation model and alignment gives a soft alignment between source side and target side. However, some of its flaws are observed under certain settings (Koehn and Knowles, 2017). One of the most common and im- portant issue of neural machine translation is that it often generates fluent but inadequate translations especially under domain mismatch conditions. An example1 is shown in Figure 1. Here, the translation generated by the NMT models — while being fluent English — has no semantic connec- tion to the source sentence. Moreover, out-of- domain models cause even more severe inade- quacy. An intuitive explanation for this observation is that the NMT decoder lacks effective attention to the source information. Because of the similar- ity between the NMT decoder and an RNNLM, it is possible that NMT models generate sentence based on its internal language model without prop- erly taking advantage of the source information. While several researchers have explored the atten- 1In this example, we choose a sentence that has been pro- cessed by byte …

Content truncated to 500 words