One Sentence One Model for Neural Machine Translation

One Sentence One Model for Neural Machine Translation Xiaoqing Li, Jiajun Zhang, Chengqing Zong National Laboratory of Pattern Recognition, CASIA, Beijing, China {xqli, jjzhang, cqzong}@nlpr.ia.ac.cn Abstract Neural machine translation (NMT) becomes a new state of the art and achieves promising translation performance using a simple encoder-decoder neural network. This neural network is trained once on the parallel corpus and the fixed network is used to translate all the test sentences. We argue that the general fixed network parameters cannot best fit each specific testing sentences. In this paper, we propose the dynamic NMT which learns a general network as usual, and then fine-tunes the network for each test sentence. The fine-tune work is done on a small set of the bilingual training data that is obtained through similarity search according to the test sentence. Extensive experiments demonstrate that this method can significantly improve the translation performance, especially when highly similar sentences are available. Keywords: Neural machine translation, online learning, sentence similarity 1. Introduction Neural machine translation achieved great success recently (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015). Thanks to the end-to-end train- ing paradigm and the powerful modeling capacity of neu- ral network, NMT can produce comparable or even bet- ter results than traditional statistical machine translation, only after a few years of development. However, it also raises some new problems, such as how to use open vocab- ulary and how to avoid repeating and missing translations. These problems have been addressed by various recent ap- proaches (Luong et al., 2015b; Jean et al., 2015; Tu et al., 2016; Mi et al., 2016). How to learn a good set of parameters is another challenge for nowadays deep neural networks. There has been some work in the field of NMT. Shen et al. (2015) propose to use task specific optimization function. Specially, they propose to directly optimize BLEU score instead of likelihood of the training data. Bengio et al. (2015) take search into con- sideration during training. In common practice, the decoder uses gold reference as history during training, but it has to use generated output as history during testing. To fix this discrepancy between training and testing, the authors pro- pose to moderately replace gold reference with generated output during training. Wiseman and Rush (2016) take a similar approach and regard training as beam search opti- mization. However, no matter how the network parameters are learnt, they are fixed after the training is finished in all current NMT practice. And the same model is applied to every testing sentence. A potential issue of this practice is that a neural network needs to be able to compress all translation knowledge into a fixed set of parameters, which is very hard in reality. So we propose to learn a specific model for each testing sentence by paying more attention to those related sentences. In particular, we propose a learning on-the-fly strategy for parameter fine-tuning. First, a general model is learnt from the whole training data. Then, for …

Content truncated to 500 words