Strategies for Training Large Vocabulary Neural Language Models

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1975–1985, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Strategies for Training Large Vocabulary Neural Language Models Wenlin Chen David Grangier Michael Auli Facebook, Menlo Park, CA Abstract Training neural network language mod- els over large vocabularies is computa- tionally costly compared to count-based models such as Kneser-Ney. We present a systematic comparison of neural strate- gies to represent and train large vocabular- ies, including softmax, hierarchical soft- max, target sampling, noise contrastive es- timation and self normalization. We ex- tend self normalization to be a proper esti- mator of likelihood and introduce an effi- cient variant of softmax. We evaluate each method on three popular benchmarks, ex- amining performance on rare words, the speed/accuracy trade-off and complemen- tarity to Kneser-Ney. 1 Introduction Neural network language models (Bengio et al., 2003; Mikolov et al., 2010) have gained popular- ity for tasks such as automatic speech recognition (Arisoy et al., 2012) and statistical machine trans- lation (Schwenk et al., 2012; Vaswani et al., 2013; Baltescu and Blunsom, 2014). Similar models are also developed for translation (Le et al., 2012; De- vlin et al., 2014; Bahdanau et al., 2015), summa- rization (Chopra et al., 2015) and language gener- ation (Sordoni et al., 2015). Language models assign a probability to a word given a context of preceding, and possibly sub- sequent, words. The model architecture deter- mines how the context is represented and there are several choices including recurrent neural net- works (Mikolov et al., 2010; Jozefowicz et al., 2016), or log-bilinear models (Mnih and Hinton, 2010). This paper does not focus on architec- ture or context representation but rather on how to efficiently deal with large output vocabularies, a problem common to all approaches to neural lan- guage modeling and related tasks (machine trans- lation, language generation). We therefore experi- ment with a classical feed-forward neural network model similar to Bengio et al. (2003). Practical training speed for these models quickly decreases as the vocabulary grows. This is due to three combined factors: (i) model evaluation and gradient computation become more time con- suming, mainly due to the need of computing nor- malized probabilities over a large vocabulary; (ii) large vocabularies require more training data in order to observe enough instances of infrequent words which increases training times; (iii) a larger training set often allows for larger models which requires more training iterations. This paper provides an overview of popular strategies to model large vocabularies for language modeling. This includes the classical softmax over all output classes, hierarchical softmax which in- troduces latent variables, or clusters, to simplify normalization, target sampling which only con- siders a random subset of classes for normaliza- tion, noise contrastive estimation which discrim- inates between genuine data points and samples from a noise distribution, and infrequent normal- ization, also referred as self-normalization, which computes the partition function at an infrequent rate. We also extend self-normalization to be a proper estimator of likelihood. …

Content truncated to 500 words