Addressing the Rare Word Problem in Neural Machine Translation

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 11–19, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Addressing the Rare Word Problem in Neural Machine Translation Minh-Thang Luong† ∗ Stanford lmthang@stanford.edu Ilya Sutskever† Google Quoc V. Le† Google {ilyasu,qvl,vinyals}@google.com Oriol Vinyals Google Wojciech Zaremba∗ New York University woj.zaremba@gmail.com Abstract Neural Machine Translation (NMT) is a new approach to machine translation that has shown promising results that are com- parable to traditional approaches. A sig- nificant weakness in conventional NMT systems is their inability to correctly trans- late very rare words: end-to-end NMTs tend to have relatively small vocabularies with a single unk symbol that represents every possible out-of-vocabulary (OOV) word. In this paper, we propose and im- plement an effective technique to address this problem. We train an NMT system on data that is augmented by the output of a word alignment algorithm, allowing the NMT system to emit, for each OOV word in the target sentence, the position of its corresponding word in the source sen- tence. This information is later utilized in a post-processing step that translates every OOV word using a dictionary. Our exper- iments on the WMT’14 English to French translation task show that this method pro- vides a substantial improvement of up to 2.8 BLEU points over an equivalent NMT system that does not use this technique. With 37.5 BLEU points, our NMT sys- tem is the first to surpass the best result achieved on a WMT’14 contest task. 1 Introduction Neural Machine Translation (NMT) is a novel ap- proach to MT that has achieved promising results (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2015; Jean et al., 2015). An NMT system is a conceptu- ally simple large neural network that reads the en- ∗Work done while the authors were in Google. † indicates equal contribution. tire source sentence and produces an output trans- lation one word at a time. NMT systems are ap- pealing because they use minimal domain knowl- edge which makes them well-suited to any prob- lem that can be formulated as mapping an input sequence to an output sequence (Sutskever et al., 2014). In addition, the natural ability of neural networks to generalize implies that NMT systems will also generalize to novel word phrases and sen- tences that do not occur in the training set. In addi- tion, NMT systems potentially remove the need to store explicit phrase tables and language models which are used in conventional systems. Finally, the decoder of an NMT system is easy to imple- ment, unlike the highly intricate decoders used by phrase-based systems (Koehn et al., 2003). Despite these advantages, conventional NMT systems are incapable of translating rare words be- cause they have a fixed modest-sized vocabulary1 which forces them to use the unk symbol to repre- sent the large number of out-of-vocabulary (OOV) words, as illustrated in Figure 1. …

Content truncated to 500 words