Simple, Fast Noise-Contrastive Estimation for Large RNN Vocabularies

Proceedings of NAACL-HLT 2016, pages 1217–1222, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Simple, Fast Noise-Contrastive Estimation for Large RNN Vocabularies Barret Zoph∗, Ashish Vaswani∗, Jonathan May, and Kevin Knight Information Sciences Institute Department of Computer Science University of Southern California {zoph, avaswani, jonmay, knight}@isi.edu Abstract We present a simple algorithm to efficiently train language models with noise-contrastive estimation (NCE) on graphics processing units (GPUs). Our NCE-trained language models achieve significantly lower perplexity on the One Billion Word Benchmark language modeling challenge, and contain one sixth of the parameters in the best single model in Chelba et al. (2013). When incorporated into a strong Arabic-English machine translation system they give a strong boost in translation quality. We release a toolkit so that others may also train large-scale, large vocabulary LSTM language models with NCE, paralleliz- ing computation across multiple GPUs. 1 Introduction Language models are used to compute probabili- ties of sequences of words. They are crucial for good performance in tasks like machine translation, speech recognition, and spelling correction. They can be classified into two categories: count-based and continuous-space language models. The lan- guage modeling literature abounds with success- ful approaches for learning-count based language models: modified Kneser-Ney smoothing, Jelinek- Mercer smoothing, etc. In recent years, continuous- space language models such as feed-forward neu- ral probabilistic language models (NPLMs) and re- current neural network language models (RNNs)1 ∗Equal contribution. 1Henceforth we will use terms like ”RNN” and ”LSTM” with the understanding that we are referring to language models that use these formalisms have outperformed their count-based counterparts (Chelba et al., 2013; Zaremba et al., 2014; Mikolov, 2012). RNNs are more powerful than n-gram lan- guage models, as they can exploit longer word con- texts to predict words. Long short-term memory lan- guage models (LSTMs) are a class of RNNs that have been designed to model long histories and are easier to train than standard RNNs. LSTMs are cur- rently the best performing language models on the Penn Treebank (PTB) dataset (Zaremba et al., 2014). The most common method for training LSTMs, maximum likelihood estimation (MLE), is pro- hibitively expensive for large vocabularies, as it in- volves time-intensive matrix-matrix multiplications. Noise-contrastive estimation (NCE) has been a suc- cessful alternative to train continuous space lan- guage models with large vocabularies (Mnih and Teh, 2012; Vaswani et al., 2013). However, NCE in its standard form is not suitable for GPUs, as the computations are not amenable to dense ma- trix operations. In this paper, we present a natu- ral modification to the NCE objective function for language modeling that allows a very efficient GPU implementation. Using our new objective, we train large multi-layer LSTMs on the One Billion Word benchmark (Chelba et al., 2013), with its full 780k word vocabulary. We achieve significantly lower perplexities with a single model, while using only a sixth of the parameters of a very strong base- line model (Chelba et al., 2013). We release our toolkit2 to allow researchers to train large-scale, …

Content truncated to 500 words