Pre-Computable Multi-Layer Neural Network Language Models

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 256–260, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Pre-Computable Multi-Layer Neural Network Language Models Jacob Devlin Microsoft Research Redmond, WA, USA jdevlin@microsoft.com Chris Quirk Microsoft Research Redmond, WA, USA chrisq@microsoft.com Arul Menezes Microsoft Research Redmond, WA, USA arulm@microsoft.com Abstract In the last several years, neural network models have significantly improved accu- racy in a number of NLP tasks. How- ever, one serious drawback that has im- peded their adoption in production sys- tems is the slow runtime speed of neu- ral network models compared to alternate models, such as maximum entropy classi- fiers. In Devlin et al. (2014), the authors presented a simple technique for speeding up feed-forward embedding-based neural network models, where the dot product be- tween each word embedding and part of the first hidden layer are pre-computed of- fline. However, this technique cannot be used for hidden layers beyond the first. In this paper, we explore a neural network architecture where the embedding layer feeds into multiple hidden layers that are placed “next to” one another so that each can be pre-computed independently. On a large scale language modeling task, this architecture achieves a 10x speedup at run- time and a significant reduction in perplex- ity when compared to a standard multi- layer network. 1 Introduction Neural network models have become extremely popular in the last several years for a wide va- riety of NLP tasks, including language model- ing (Schwenk, 2007), sentiment analysis (Socher et al., 2013), translation modeling (Devlin et al., 2014), and many others (Collobert et al., 2011). However, a serious drawback of neural network models is their slow speeds in training and test time (runtime) relative to alternative models such as maximum entropy (Berger et al., 1996) or back- off models (Kneser and Ney, 1995). One popular application of neural network models in NLP is using neural network language models (NNLMs) as an additional feature in an existing machine translation (MT) or automatic speech recognition (ASR) engines. NNLMs are particularly costly in this scenario, since decoding a single sentence typically requires tens of thou- sands or more n-gram lookups. Although we will focus on this particular scenario in this paper, it is important to note that the techniques presented generalize to any feed-forward embedding-based neural network model. One popular technique for improving the run- time speed of NNLMs involves training the net- work to be “approximately normalized,” so that the softmax normalizer does not have to be com- puted after training. Two algorithms have been proposed to achieve this: (1) noise-contrastive es- timation (NCE) (Mnih and Teh, 2012; Vaswani et al., 2013) and (2) explicit self-normalization (De- vlin et al., 2014), which is used in this paper. However, even with self-normalized networks, computing the output of an intermediate hidden layer still requires a costly matrix-vector multipli- cation. To mitigate this, Devlin et al. (2014) made the observation that for 1-layer NNLMs, the dot product between each embedding+position …

Content truncated to 500 words