On Using Very Large Target Vocabulary for Neural Machine Translation

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1–10, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics On Using Very Large Target Vocabulary for Neural Machine Translation S´ebastien Jean Kyunghyun Cho Roland Memisevic Universit´e de Montr´eal Yoshua Bengio Universit´e de Montr´eal CIFAR Senior Fellow Abstract Neural machine translation, a recently proposed approach to machine transla- tion based purely on neural networks, has shown promising results compared to the existing approaches such as phrase- based statistical machine translation. De- spite its recent success, neural machine translation has its limitation in handling a larger vocabulary, as training complex- ity as well as decoding complexity in- crease proportionally to the number of tar- get words. In this paper, we propose a method based on importance sampling that allows us to use a very large target vo- cabulary without increasing training com- plexity. We show that decoding can be efficiently done even with the model hav- ing a very large target vocabulary by se- lecting only a small subset of the whole target vocabulary. The models trained by the proposed approach are empirically found to match, and in some cases out- perform, the baseline models with a small vocabulary as well as the LSTM-based neural machine translation models. Fur- thermore, when we use an ensemble of a few models with very large target vo- cabularies, we achieve performance com- parable to the state of the art (measured by BLEU) on both the English→German and English→French translation tasks of WMT’14. 1 Introduction Neural machine translation (NMT) is a recently introduced approach to solving machine transla- tion (Kalchbrenner and Blunsom, 2013; Bahdanau et al., 2015; Sutskever et al., 2014). In neural ma- chine translation, one builds a single neural net- work that reads a source sentence and generates its translation. The whole neural network is jointly trained to maximize the conditional probability of a correct translation given a source sentence, us- ing the bilingual corpus. The NMT models have shown to perform as well as the most widely used conventional translation systems (Sutskever et al., 2014; Bahdanau et al., 2015). Neural machine translation has a number of advantages over the existing statistical machine translation system, specifically, the phrase-based system (Koehn et al., 2003). First, NMT requires a minimal set of domain knowledge. For instance, all of the models proposed in (Sutskever et al., 2014), (Bahdanau et al., 2015) or (Kalchbrenner and Blunsom, 2013) do not assume any linguis- tic property in both source and target sentences except that they are sequences of words. Sec- ond, the whole system is jointly trained to maxi- mize the translation performance, unlike the exist- ing phrase-based system which consists of many separately trained features whose weights are then tuned jointly. Lastly, the memory footprint of the NMT model is often much smaller than the exist- ing system which relies on maintaining large ta- bles of phrase pairs. Despite these advantages and promising results, …

Content truncated to 500 words