Shrinking Exponential Language Models

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 468–476, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Shrinking Exponential Language Models Stanley F. Chen IBM T.J. Watson Research Center P.O. Box 218, Yorktown Heights, NY 10598 stanchen@watson.ibm.com Abstract In (Chen, 2009), we show that for a vari- ety of language models belonging to the ex- ponential family, the test set cross-entropy of a model can be accurately predicted from its training set cross-entropy and its parameter values. In this work, we show how this rela- tionship can be used to motivate two heuristics for “shrinking” the size of a language model to improve its performance. We use the first heuristic to develop a novel class-based lan- guage model that outperforms a baseline word trigram model by 28% in perplexity and 1.9% absolute in speech recognition word-error rate on Wall Street Journal data. We use the second heuristic to motivate a regularized version of minimum discrimination information models and show that this method outperforms other techniques for domain adaptation. 1 Introduction An exponential model pΛ(y|x) is a model with a set of features {f1(x, y), . . . , fF (x, y)} and equal num- ber of parameters Λ = {λ1, . . . , λF } where pΛ(y|x) = exp(�F i=1 λifi(x, y)) ZΛ(x) (1) and where ZΛ(x) is a normalization factor. In (Chen, 2009), we show that for many types of ex- ponential language models, if a training and test set are drawn from the same distribution, we have Htest ≈ Htrain + γ D F � i=1 |˜λi| (2) where Htest denotes test set cross-entropy; Htrain de- notes training set cross-entropy; D is the number of events in the training data; the ˜λi are regularized pa- rameter estimates; and γ is a constant independent of domain, training set size, and model type.1 This relationship is strongest if the ˜Λ = {˜λi} are esti- mated using ℓ1+ℓ2 2 regularization (Kazama and Tsu- jii, 2003). In ℓ1 + ℓ2 2 regularization, parameters are chosen to optimize Oℓ1+ℓ2 2(Λ) = Htrain + α D F � i=1 |λi| + 1 2σ2D F � i=1 λ2 i (3) for some α and σ. With (α = 0.5, σ2 = 6) and taking γ = 0.938, test set cross-entropy can be pre- dicted with eq. (2) for a wide range of models with a mean error of a few hundredths of a nat, equivalent to a few percent in perplexity.2 In this paper, we show how eq. (2) can be applied to improve language model performance. First, we use eq. (2) to analyze backoff features in exponential n-gram models. We find that backoff features im- prove test set performance by reducing the “size” of a model 1 D �F i=1 |˜λi| rather than by improving train- ing set performance. This suggests the following principle for improving exponential language mod- els: if a model can be “shrunk” without increasing its training set cross-entropy, test set …

Content truncated to 500 words