Exponential Priors for Maximum Entropy Models

Exponential Priors for Maximum Entropy Models Joshua Goodman One Microsoft Way Redmond, WA 98052 joshuago@microsoft.com Abstract Maximum entropy models are a common mod- eling technique, but prone to overfitting. We show that using an exponential distribution as a prior leads to bounded absolute discounting by a constant. We show that this prior is better motivated by the data than previous techniques such as a Gaussian prior, and often produces lower error rates. Exponential priors also lead to a simpler learning algorithm and to easier to understand behavior. Furthermore, exponential priors help explain the success of some previ- ous smoothing techniques, and suggest simple variations that work better. 1 Introduction Conditional Maximum Entropy (maxent) models have been widely used for a variety of tasks, including lan- guage modeling (Rosenfeld, 1994), part-of-speech tag- ging, prepositional phrase attachment, and parsing (Rat- naparkhi, 1998), word selection for machine translation (Berger et al., 1996), and finding sentence boundaries (Reynar and Ratnaparkhi, 1997). They are also some- times called logistic regression models, maximum likeli- hood exponential models, log-linear models, and are even equivalent to a form of perceptrons/single layer neural networks. In particular, perceptrons that use the standard sigmoid function, and optimize for log-loss are equivalent to maxent. Multi-layer neural networks that optimize log- loss are closely related, and much of the discussion will apply to them implicitly. Conditional maxent models have traditionally either been unregularized or regularized by using a Gaussian prior on the parameters. We will show that by using an exponential distribution as the prior, several advantages can be gained. We will show that in at least one case, an exponential prior experimentally better matches the ac- tual distribution of the parameters. We will also show that it can lead to improved accuracy, and a simpler learning algorithm. In addition, the exponential prior inspires an improved version of Good Turing discounting with lower perplexity. Conditional maxent models are of the form PΛ(y|x) = exp �F i=1 λifi(x, y) � y′ exp � i λifi(x, y′) where x is an input vector, y is an output, the fi are the so- called indicator functions or feature values that are true if a particular property of x, y is true, F is the number of such features, Λ represents the parameter set λ1...λn, and λi is a weight for the indicator fi. For instance, if trying to do word sense disambiguation for the word “bank”, x would be the context around an occurrence of the word; y would be a particular sense, e.g. financial or river; fi(x, y) could be 1 if the context includes the word “money” and y is the financial sense; and λi would be a large positive number. Maxent models have several valuable properties (Della Pietra et al. (1997) give a good overview.) The most important is constraint satisfaction. For a given fi, we can count how many times fi was observed in the training data with value y, observed[i] = � j fi(xj, yj). For a model PΛ with parameters …

Content truncated to 500 words