Training Neural Network Language Models on Very Large Corpora

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 201–208, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Training Neural Network Language Models On Very Large Corpora ∗ Holger Schwenk and Jean-Luc Gauvain LIMSI-CNRS BP 133, 91436 Orsay cedex, FRANCE schwenk,gauvain@limsi.fr Abstract During the last years there has been grow- ing interest in using neural networks for language modeling. In contrast to the well known back-off n-gram language models, the neural network approach attempts to overcome the data sparseness problem by performing the estimation in a continuous space. This type of language model was mostly used for tasks for which only a very limited amount of in-domain training data is available. In this paper we present new algorithms to train a neural network language model on very large text corpora. This makes pos- sible the use of the approach in domains where several hundreds of millions words of texts are available. The neural network language model is evaluated in a state-of- the-art real-time continuous speech recog- nizer for French Broadcast News. Word error reductions of 0.5% absolute are re- ported using only a very limited amount of additional processing time. 1 Introduction Language models play an important role in many applications like character and speech recognition, machine translation and information retrieval. Sev- eral approaches have been developed during the last ∗This work was partially financed by the European Commis- sion under the FP6 Integrated Project TC-STAR. decades like n-gram back-off word models (Katz, 1987), class models (Brown et al., 1992), structured language models (Chelba and Jelinek, 2000) or max- imum entropy language models (Rosenfeld, 1996). To the best of our knowledge word and class n-gram back-off language models are still the dominant ap- proach, at least in applications like large vocabulary continuous speech recognition or statistical machine translation. In many publications it has been re- ported that modified Kneser-Ney smoothing (Chen and Goodman, 1999) achieves the best results. All the reference back-off language models (LM) de- scribed in this paper are build with this technique, using the SRI LM toolkit (Stolcke, 2002). The field of natural language processing has re- cently seen some changes by the introduction of new statistical techniques that are motivated by success- ful approaches from the machine learning commu- nity, in particular continuous space LMs using neu- ral networks (Bengio and Ducharme, 2001; Bengio et al., 2003; Schwenk and Gauvain, 2002; Schwenk and Gauvain, 2004; Emami and Jelinek, 2004), Ran- dom Forest LMs (Xu and Jelinek, 2004) and Ran- dom cluster LMs (Emami and Jelinek, 2005). Usu- ally new approaches are first verified on small tasks using a limited amount of LM training data. For instance, experiments have been performed using the Brown corpus (1.1M words), parts of the Wall- street journal corpus (19M words) or transcriptions of acoustic training data (up to 22M words). It is much more challenging to compare the new statis- tical techniques to carefully optimized back-off LM trained on large amounts of data …

Content truncated to 500 words