Random Forests in Language Modelin

Random Forests in Language Modeling Peng Xu and Frederick Jelinek Center for Language and Speech Processing the Johns Hopkins University Baltimore, MD 21218, USA � xp,jelinek � @jhu.edu Abstract In this paper, we explore the use of Random Forests (RFs) (Amit and Geman, 1997; Breiman, 2001) in language modeling, the problem of predicting the next word based on words already seen before. The goal in this work is to develop a new language mod- eling approach based on randomly grown Decision Trees (DTs) and apply it to automatic speech recog- nition. We study our RF approach in the context of lar� -gram type language modeling. Unlike regu- � -gram language models, RF language models have the potential to generalize well to unseen data, even when a complicated history is used. We show that our RF language models are superior to regular � -gram language models in reducing both the per- plexity (PPL) and word error rate (WER) in a large vocabulary speech recognition system. 1 Introduction In many systems dealing with natural speech or lan- guage, such as Automatic Speech Recognition and Statistical Machine Translation, a language model is a crucial component for searching in the often prohibitively large hypothesis space. Most state-of- the-art systems use � -gram language models, which are simple and effective most of the time. Many smoothing techniques that improve language model probability estimation have been proposed and stud- ied in the � -gram literature (Chen and Goodman, 1998). There has also been work in exploring Deci- sion Tree (DT) language models (Bahl et al., 1989; Potamianos and Jelinek, 1998), which attempt to cluster similar histories together to achieve better probability estimation. However, the results were not promising (Potamianos and Jelinek, 1998): in a fair comparison, decision tree language models failed to improve upon the baseline � -gram models with the same order � . The aim of DT language models is to alleviate the data sparseness problem encountered in � -gram language models. However, the cause of the neg- ative results is exactly the same: data sparseness, coupled with the fact that the DT construction al- gorithms decide on tree splits solely on the basis of seen data (Potamianos and Jelinek, 1998). Al- though various smoothing techniques were studied in the context of DT language models, none of them resulted in significant improvements over � -gram models. Recently, a neural network based language mod- eling approach has been applied to trigram lan- guage models to deal with the curse of dimension- ality (Bengio et al., 2001; Schwenk and Gauvain, 2002). Significant improvements in both perplex- ity (PPL) and word error rate (WER) over backoff smoothing were reported after interpolating the neu- ral network models with the baseline backoff mod- els. However, the neural network models rely on interpolation with � -gram models, and use � -gram models exclusively for low frequency words. We believe improvements in � -gram models should also improve the performance of neural network models. We propose a new Random Forest (RF) …

Content truncated to 500 words