Neural Machine Translation with Word Predictions

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 136–145 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Neural Machine Translation with Word Predictions Rongxiang Weng, Shujian Huang, Zaixiang Zheng, Xinyu Dai and Jiajun Chen State Key Laboratory for Novel Software Technology Nanjing University Nanjing 210023, China {wengrx, huangsj, zhengzx, daixy, chenjj}@nlp.nju.edu.cn Abstract In the encoder-decoder architecture for neural machine translation (NMT), the hidden states of the recurrent structures in the encoder and decoder carry the crucial information about the sentence.These vec- tors are generated by parameters which are updated by back-propagation of translation errors through time. We argue that prop- agating errors through the end-to-end re- current structures are not a direct way of control the hidden vectors. In this paper, we propose to use word predictions as a mechanism for direct supervision. More specifically, we require these vectors to be able to predict the vocabulary in tar- get sentence. Our simple mechanism en- sures better representations in the encoder and decoder without using any extra data or annotation. It is also helpful in reduc- ing the target side vocabulary and improv- ing the decoding efficiency. Experiments on Chinese-English and German-English machine translation tasks show BLEU im- provements by 4.53 and 1.3, respectively. 1 Introduction The encoder-decoder based neural machine trans- lation (NMT) models (Sutskever et al., 2014; Cho et al., 2014) have been developing rapidly. Sutskever et al. (2014) propose to encode the source sentence as a fixed-length vector repre- sentation, based on which the decoder gener- ates the target sequence, where both the en- coder and decoder are recurrent neural net- works (RNN) (Sutskever et al., 2014) or their vari- ants (Cho et al., 2014; Chung et al., 2014; Bah- danau et al., 2014). In this framework, the fixed- length vector plays the crucial role of transition- ing the information of the sentence from the source side to the target side. Later, attention mechanisms are proposed to en- hance the source side representations (Bahdanau et al., 2014; Luong et al., 2015b). The source side context is computed at each time-step of decod- ing, based on the attention weights between the source side representations and the current hidden state of the decoder. However, the hidden states in the recurrent decoder still originate from the single fixed-length representation (Luong et al., 2015b), or the average of the bi-directional repre- sentations (Bahdanau et al., 2014). Here we refer to the representation as initial state. Interestingly, Britz et al. (2017) find that the value of initial state does not affect the translation performance, and prefer to set the initial state to be a zero vector. On the contrary, we argue that initial state still plays an important role of transla- tion, which is currently neglected. We notice that beside the end-to-end error back propagation for the initial and transition parameters, there is no di- rect control of the initial state in the current NMT architectures. Due to the large number of param- eters, it may …

Content truncated to 500 words