A Binarized Neural Network Joint Model for Machine Translation

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2094–2099, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. A Binarized Neural Network Joint Model for Machine Translation Jingyi Zhang1,2, Masao Utiyama1, Eiichro Sumita1 Graham Neubig2, Satoshi Nakamura2 1National Institute of Information and Communications Technology, 3-5Hikaridai, Keihanna Science City, Kyoto 619-0289, Japan 2Graduate School of Information Science, Nara Institute of Science and Technology, Takayama, Ikoma, Nara 630-0192, Japan jingyizhang/mutiyama/eiichiro.sumita@nict.go.jp neubig/s-nakamura@is.naist.jp Abstract The neural network joint model (NNJM), which augments the neural network lan- guage model (NNLM) with an m-word source context window, has achieved large gains in machine translation accuracy, but also has problems with high normalization cost when using large vocabularies. Train- ing the NNJM with noise-contrastive es- timation (NCE), instead of standard maxi- mum likelihood estimation (MLE), can re- duce computation cost. In this paper, we propose an alternative to NCE, the bina- rized NNJM (BNNJM), which learns a bi- nary classifier that takes both the context and target words as input, and can be ef- ficiently trained using MLE. We compare the BNNJM and NNJM trained by NCE on various translation tasks. 1 Introduction Neural network translation models, which learn mappings over real-valued vector representations in high-dimensional space, have recently achieved large gains in translation accuracy (Hu et al., 2014; Devlin et al., 2014; Sundermeyer et al., 2014; Auli et al., 2013; Schwenk, 2012; Sutskever et al., 2014; Bahdanau et al., 2015). Notably, Devlin et al. (2014) proposed a neural network joint model (NNJM), which augments the n-gram neural network language model (NNLM) with an m-word source context window, as shown in Figure 1a. While this model is effective, the computation cost of using it in a large-vocabulary SMT task is quite expensive, as probabilities need to be normalized over the entire vocabulary. To solve this problem, Devlin et al. (2014) pre- sented a technique to train the NNJM to be self- normalized and avoided the expensive normaliza- tion cost during decoding. However, they also P(ti=1) P(ti=2) P(ti=N) ti-n+1~ti-1 m-word source context (a) P(ti is correct) P(ti is wrong) ti-n+1~ti source context m-word (b) Figure 1: (a) the traditional NNJM and (b) the pro- posed BNNJM note that this self-normalization technique sacri- fices neural network accuracy, and the training process for the self-normalized neural network is very slow, as with standard maximum likelihood estimation (MLE). To remedy the problem of long training times in the context of NNLMs, Vaswani et al. (2013) used a method called noise contrastive estimation (NCE). Compared with MLE, NCE does not re- quire repeated summations over the whole vocab- ulary and performs nonlinear logistic regression to discriminate between the observed data and artifi- cially generated noise. This paper proposes an alternative framework of binarized NNJMs (BNNJM), which are similar to the NNJM, but use the current target word not as the output, but as the input of the neural network, estimating whether the target word under exam- ination is correct or not, as shown in Figure 1b. …

Content truncated to 500 words