An Empirical Evaluation of Noise Contrastive Estimation for the Neural Network Joint Model of Translation

Proceedings of NAACL-HLT 2016, pages 41–46, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics An Empirical Evaluation of Noise Contrastive Estimation for the Neural Network Joint Model of Translation Colin Cherry National Research Council Canada Colin.Cherry@nrc-cnrc.gc.ca Abstract The neural network joint model of transla- tion or NNJM (Devlin et al., 2014) combines source and target context to produce a power- ful translation feature. However, its softmax layer necessitates a sum over the entire output vocabulary, which results in very slow max- imum likelihood (MLE) training. This has led some groups to train using Noise Con- trastive Estimation (NCE), which side-steps this sum. We carry out the first direct compar- ison of MLE and NCE training objectives for the NNJM, showing that NCE is significantly outperformed by MLE on large-scale Arabic- English and Chinese-English translation tasks. We also show that this drop can be avoided by using a recently proposed translation noise distribution. 1 Introduction The Neural Network Joint Model of Translation, or NNJM (Devlin et al., 2014), is a strong feature for statistical machine translation. The NNJM uses both target and source tokens as context for a feed- forward neural network language model (LM). Un- fortunately, its softmax layer requires a sum over the entire output vocabulary, which slows the calcula- tion of LM probabilities and the maximum likeli- hood estimation (MLE) of model parameters. Devlin et al. (2014) address this problem at run- time only with a self-normalized MLE objective. Others advocate the use of Noise Contrastive Esti- mation (NCE) to train NNJMs and similar mono- lingual LMs (Mnih and Teh, 2012; Vaswani et al., 2013; Baltescu and Blunsom, 2015; Zhang et al., 2015). NCE avoids the sum over the output vo- cabulary at both train- and run-time by wrapping the NNJM inside a classifier that attempts to sep- arate real data from sampled noise, greatly im- proving training speed. The training efficiency of NCE is well-documented, and will not be evaluated here. However, the experimental evidence that NCE matches MLE in terms of resulting model quality is all on monolingual language modeling tasks (Mnih and Teh, 2012). Since cross-lingual contexts pro- vide substantially stronger signals than monolingual ones, there is reason to suspect these results may not carry over to NNJMs. To our knowledge there is no published work that directly compares MLE and NCE in the context of an NNJM; this paper fills that gap as its primary con- tribution. We measure model likelihood and trans- lation quality in large-scale Arabic-to-English and Chinese-to-English translation tasks. We also test a recently-proposed translation noise distribution for NCE (Zhang et al., 2015), along with a mixture of noise distributions. Finally, we test a widely known, but apparently undocumented, technique for domain adaptation of NNJMs, demonstrating its utility, as well as its impact on the MLE-NCE comparison. 2 Methods The NNJM adds a bilingual context window to the machinery of feed-forward neural network language models, or NNLMs (Bengio et al., 2003). An NNLM calculates the probability p(ei|ei−1 i−n+1) …

Content truncated to 500 words