Adaptive Weighting for Neural Machine Translation

Proceedings of the 27th International Conference on Computational Linguistics, pages 3038–3048 Santa Fe, New Mexico, USA, August 20-26, 2018. 3038 Adaptive Weighting for Neural Machine Translation Yachao Li1,2, Junhui Li1, Min Zhang1,∗ 1Institute of Artificial Intelligence, School of Computer Science and Technology, Soochow University, China 2Key Laboratory of China’s Ethnic Languages and Information Technology of Ministry of Education, Northwest Minzu University, China liyc7711@gmail.com, {lijunhui, minzhang}@suda.edu.cn Abstract In the popular sequence to sequence (seq2seq) neural machine translation (NMT), there exist many weighted sum models (WSMs), each of which takes a set of input and generates one output. However, the weights in a WSM are independent of each other and fixed for all inputs, suggesting that by ignoring different needs of inputs, the WSM lacks effective control on the influence of each input. In this paper, we propose adaptive weighting for WSMs to control the contribution of each input. Specifically, we apply adaptive weighting for both GRU and the output state in NMT. Experimentation on Chinese-to-English translation and English-to-German translation demonstrates that the proposed adaptive weighting is able to much improve translation accuracy by achieving significant improvement of 1.49 and 0.92 BLEU points for the two translation tasks. Moreover, we discuss in-depth on what type of information is encoded in the encoder and how information influences the generation of target words in the decoder. 1 Introduction Recent advances in neural machine translation (NMT) have achieved remarkable success over the state- of-the-art of statistical machine translation (SMT) on various language pairs (Bahdanau et al., 2015; Jean et al., 2015; Luong et al., 2015; Wu et al., 2016; Vaswani et al., 2017). In the neural networks of seq2seq models, either RNN-based (Bahdanau et al., 2015), CNN-based (Gehring et al., 2017), or full attention-based (Vaswani et al., 2017), there exist many scenarios in both encoder and decoder where a weighted sum model (WSM) takes a set of inputs and generate one output. As shown in Eq. 1, the WSM first combines k inputs (x1, · · · , xk) with k respective weights (w1, · · · , wk) and then non-linearizes it through an activation function f, such as tanh, sigmoid function, ReLU, and so on. In this paper we omit bias terms to make the equations less cluttered. o = f � k � i=1 wixi � (1) Note that the above weights (w1, · · · , wk) are independent of each other and once the model is tuned, the weights are fixed for all inputs, suggesting that by ignoring different needs of the inputs, the WSM lacks effective control on the influence of each input. Let us take a concrete scenario in seq2seq model as an example. Figure 1(a) shows a typical illustration of generation of t-th target word where the decoder takes three inputs, i.e., source context ct, previous target word yt−1 and current target state st while generating one output yt via output state ot. However, the study in Tu et al. (2017) suggests that different target words require …

Content truncated to 500 words