Training Continuous Space Language Models: Some Practical Issues

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 778–788, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Training continuous space language models: some practical issues Le Hai Son and Alexandre Allauzen and Guillaume Wisniewski and Franc¸ois Yvon Univ. Paris-Sud, France and LIMSI/CNRS BP 133, 91403 Orsay Cedex Firstname.Lastname@limsi.fr Abstract Using multi-layer neural networks to esti- mate the probabilities of word sequences is a promising research area in statistical lan- guage modeling, with applications in speech recognition and statistical machine transla- tion. However, training such models for large vocabulary tasks is computationally challeng- ing which does not scale easily to the huge corpora that are nowadays available. In this work, we study the performance and behav- ior of two neural statistical language models so as to highlight some important caveats of the classical training algorithms. The induced word embeddings for extreme cases are also analysed, thus providing insight into the con- vergence issues. A new initialization scheme and new training techniques are then intro- duced. These methods are shown to greatly re- duce the training time and to significantly im- prove performance, both in terms of perplexity and on a large-scale translation task. 1 Introduction Statistical language models play an important role in many practical applications, such as machine trans- lation and automatic speech recognition. Let V be a finite vocabulary, statistical language models de- fine distributions over sequences of words wL 1 in V⋆ usually factorized as: P(wL 1 ) = P(w1) L � l=1 P(wl|wl−1 1 ) Modeling the joint distribution of several discrete random variables (such as words in a sentence) is difficult, especially in real-world Natural Language Processing applications where V typically contains dozens of thousands words. Many approaches to this problem have been pro- posed over the last decades, the most widely used being back-off n-gram language models. n-gram models rely on a Markovian assumption, and de- spite this simplification, the maximum likelihood es- timate (MLE) remains unreliable and tends to under- estimate the probability of very rare n-grams, which are hardly observed even in huge corpora. Con- ventional smoothing techniques, such as Kneser- Ney and Witten-Bell back-off schemes (see (Chen and Goodman, 1996) for an empirical overview, and (Teh, 2006) for a Bayesian interpretation), per- form back-off on lower order distributions to pro- vide an estimate for the probability of these unseen events. n-gram language models rely on a discrete space representation of the vocabulary, where each word is associated with a discrete index. In this model, the morphological, syntactic and semantic relationships which structure the lexicon are com- pletely ignored, which negatively impact the gen- eralization performance of the model. Various ap- proaches have proposed to overcome this limita- tion, notably the use of word-classes (Brown et al., 1992; Niesler, 1997), of generalized back-off strate- gies (Bilmes et al., 1997) or the explicit integration of morphological information in the random-forest model (Xu and Jelinek, 2004; Oparin et al., 2008). One of the most successful alternative to …

Content truncated to 500 words