A Hierarchical Bayesian Language Model Based On Pitman-Yor Processes

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 985–992, Sydney, July 2006. c⃝2006 Association for Computational Linguistics A Hierarchical Bayesian Language Model based on Pitman-Yor Processes Yee Whye Teh School of Computing, National University of Singapore, 3 Science Drive 2, Singapore 117543. tehyw@comp.nus.edu.sg Abstract We propose a new hierarchical Bayesian n-gram model of natural languages. Our model makes use of a generalization of the commonly used Dirichlet distributions called Pitman-Yor processes which pro- duce power-law distributions more closely resembling those in natural languages. We show that an approximation to the hier- archical Pitman-Yor language model re- covers the exact formulation of interpo- lated Kneser-Ney, one of the best smooth- ing methods for n-gram language models. Experiments verify that our model gives cross entropy results superior to interpo- lated Kneser-Ney and comparable to mod- ified Kneser-Ney. 1 Introduction Probabilistic language models are used exten- sively in a variety of linguistic applications, in- cluding speech recognition, handwriting recogni- tion, optical character recognition, and machine translation. Most language models fall into the class of n-gram models, which approximate the distribution over sentences using the conditional distribution of each word given a context consist- ing of only the previous n − 1 words, P(sentence) ≈ T � i=1 P(wordi | wordi−1 i−n+1) (1) with n = 3 (trigram models) being typical. Even for such a modest value of n the number of param- eters is still tremendous due to the large vocabu- lary size. As a result direct maximum-likelihood parameter fitting severely overfits to the training data, and smoothing methods are indispensible for proper training of n-gram models. A large number of smoothing methods have been proposed in the literature (see (Chen and Goodman, 1998; Goodman, 2001; Rosenfeld, 2000) for good overviews). Most methods take a rather ad hoc approach, where n-gram probabili- ties for various values of n are combined together, using either interpolation or back-off schemes. Though some of these methods are intuitively ap- pealing, the main justification has always been empirical—better perplexities or error rates on test data. Though arguably this should be the only real justification, it only answers the question of whether a method performs better, not how nor why it performs better. This is unavoidable given that most of these methods are not based on in- ternally coherent Bayesian probabilistic models, which have explicitly declared prior assumptions and whose merits can be argued in terms of how closely these fit in with the known properties of natural languages. Bayesian probabilistic mod- els also have additional advantages—it is rela- tively straightforward to improve these models by incorporating additional knowledge sources and to include them in larger models in a principled manner. Unfortunately the performance of pre- viously proposed Bayesian language models had been dismal compared to other smoothing meth- ods (Nadas, 1984; MacKay and Peto, 1994). In this paper, we propose a novel language model based on a hierarchical Bayesian model (Gelman et al., 1995) where each hidden variable is …

Content truncated to 500 words