Deep Neural Language Models for Machine Translation

Proceedings of the 19th Conference on Computational Language Learning, pages 305–309, Beijing, China, July 30-31, 2015. c⃝2015 Association for Computational Linguistics Deep Neural Language Models for Machine Translation Minh-Thang Luong Michael Kayser Christopher D. Manning Computer Science Department, Stanford University, Stanford, CA, 94305 {lmthang, mkayser, manning}@stanford.edu Abstract Neural language models (NLMs) have been able to improve machine translation (MT) thanks to their ability to generalize well to long contexts. Despite recent suc- cesses of deep neural networks in speech and vision, the general practice in MT is to incorporate NLMs with only one or two hidden layers and there have not been clear results on whether having more lay- ers helps. In this paper, we demonstrate that deep NLMs with three or four lay- ers outperform those with fewer layers in terms of both the perplexity and the trans- lation quality. We combine various tech- niques to successfully train deep NLMs that jointly condition on both the source and target contexts. When reranking n- best lists of a strong web-forum baseline, our deep models yield an average boost of 0.5 TER / 0.5 BLEU points compared to using a shallow NLM. Additionally, we adapt our models to a new sms-chat do- main and obtain a similar gain of 1.0 TER / 0.5 BLEU points.1 1 Introduction Deep neural networks (DNNs) have been success- ful in learning more complex functions than shal- low ones (Bengio, 2009) and exceled in many challenging tasks such as in speech (Hinton et al., 2012) and vision (Krizhevsky et al., 2012). These results have sparked interest in applying DNNs to natural language processing problems as well. Specifically, in machine translation (MT), there 1Our code and related materials are publicly available at http://stanford.edu/˜lmthang/nlm. has been an active body of work recently in uti- lizing neural language models (NLMs) to improve translation quality. However, to the best of our knowledge, work in this direction only makes use of NLMs with either one or two hidden layers. For example, Schwenk (2010, 2012) and Son et al. (2012) used shallow NLMs with a single hidden layer for reranking. Vaswani et al. (2013) consid- ered two-layer NLMs for decoding but provided no comparison among models of various depths. Devlin et al. (2014) reported only a small gain when decoding with a two-layer NLM over a sin- gle layer one. There have not been clear results on whether adding more layers to NLMs helps. In this paper, we demonstrate that deep NLMs with three or four layers are better than those with fewer layers in terms of the perplexity and the translation quality. We detail how we com- bine various techniques from past work to suc- cessfully train deep NLMs that condition on both the source and target contexts. When reranking n- best lists of a strong web-forum MT baseline, our deep models achieve an additional improvement of 0.5 TER / 0.5 BLEU compared to using a shal- low NLM. Furthermore, by fine-tuning general in- domain NLMs with out-of-domain data, we obtain …

Content truncated to 500 words