Decoding with Large-Scale Neural Language Models Improves Translation

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1387–1392, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Decoding with Large-Scale Neural Language Models Improves Translation Ashish Vaswani University of Southern California Department of Computer Science avaswani@isi.edu Yinggong Zhao Nanjing University, State Key Laboratory for Novel Software Technology zhaoyg@nlp.nju.edu.cn Victoria Fossum and David Chiang University of Southern California Information Sciences Institute {vfossum,chiang}@isi.edu Abstract We explore the application of neural language models to machine translation. We develop a new model that combines the neural proba- bilistic language model of Bengio et al., rec- tified linear units, and noise-contrastive esti- mation, and we incorporate it into a machine translation system both by reranking k-best lists and by direct integration into the decoder. Our large-scale, large-vocabulary experiments across four language pairs show that our neu- ral language model improves translation qual- ity by up to 1.1 Bleu. 1 Introduction Machine translation (MT) systems rely upon lan- guage models (LMs) during decoding to ensure flu- ent output in the target language. Typically, these LMs are n-gram models over discrete representa- tions of words. Such models are susceptible to data sparsity–that is, the probability of an n-gram ob- served only few times is difficult to estimate reli- ably, because these models do not use any informa- tion about similarities between words. To address this issue, Bengio et al. (2003) pro- pose distributed word representations, in which each word is represented as a real-valued vector in a high-dimensional feature space. Bengio et al. (2003) introduce a feed-forward neural probabilistic LM (NPLM) that operates over these distributed repre- sentations. During training, the NPLM learns both a distributed representation for each word in the vo- cabulary and an n-gram probability distribution over words in terms of these distributed representations. Although neural LMs have begun to rival or even surpass traditional n-gram LMs (Mnih and Hin- ton, 2009; Mikolov et al., 2011), they have not yet been widely adopted in large-vocabulary applica- tions such as MT, because standard maximum like- lihood estimation (MLE) requires repeated summa- tions over all words in the vocabulary. A variety of strategies have been proposed to combat this issue, many of which require severe restrictions on the size of the network or the size of the data. In this work, we extend the NPLM of Bengio et al. (2003) in two ways. First, we use rectified lin- ear units (Nair and Hinton, 2010), whose activa- tions are cheaper to compute than sigmoid or tanh units. There is also evidence that deep neural net- works with rectified linear units can be trained suc- cessfully without pre-training (Zeiler et al., 2013). Second, we train using noise-contrastive estimation or NCE (Gutmann and Hyv¨arinen, 2010; Mnih and Teh, 2012), which does not require repeated summa- tions over the whole vocabulary. This enables us to efficiently build NPLMs on a larger scale than would be possible otherwise. We then apply this LM to MT in two ways. First, we use it to …

Content truncated to 500 words