Parametric Models of Linguistic Count Data

Parametric Models of Linguistic Count Data Martin Jansche Department of Linguistics The Ohio State University Columbus, OH 43210, USA jansche@acm.org Abstract It is well known that occurrence counts of words in documents are often mod- eled poorly by standard distributions like the binomial or Poisson. Observed counts vary more than simple models predict, prompting the use of overdispersed mod- els like Gamma-Poisson or Beta-binomial mixtures as robust alternatives. Another deficiency of standard models is due to the fact that most words never occur in a given document, resulting in large amounts of zero counts. We propose using zero- inflated models for dealing with this, and evaluate competing models on a Naive Bayes text classification task. Simple zero-inflated models can account for prac- tically relevant variation, and can be easier to work with than overdispersed models. 1 Introduction Linguistic count data often violate the simplistic as- sumptions of standard probability models like the binomial or Poisson distribution. In particular, the inadequacy of the Poisson distribution for model- ing word (token) frequency is well known, and ro- bust alternatives have been proposed (Mosteller and Wallace, 1984; Church and Gale, 1995). In the case of the Poisson, a commonly used robust alternative is the negative binomial distribution (Pawitan, 2001, §4.5), which has the ability to capture extra-Poisson variation in the data, in other words, it is overdis- persed compared with the Poisson. When a small set of parameters controls all properties of the dis- tribution it is important to have enough parameters to model the relevant aspects of one’s data. Sim- ple models like the Poisson or binomial do not have enough parameters for many realistic applications, and we suspect that the same might be true of log- linear models. When applying robust models like the negative binomial to linguistic count data like word occurrences in documents, it is natural to ask to what extent the extra-Poisson variation has been captured by the model. Answering that question is our main goal, and we begin by reviewing some of the classic results of Mosteller and Wallace (1984). 2 Word Frequency in Fixed-Length Texts In preparation of their authorship study of The Fed- eralist, Mosteller and Wallace (1984, §2.3) investi- gated the variation of word frequency across con- tiguous passages of similar length, drawn from pa- pers of known authorship. The occurrence frequen- cies of any in papers by Hamilton (op. cit., Ta- ble 2.3–3) are repeated here in Figure 1: out of a total of 247 passages there are 125 in which the word any does not occur; it occurs once in 88 pas- sages, twice in 26 passages, etc. Figure 1 also shows the counts predicted by a Poisson distribution with mean 0.67. Visual inspection (“chi by eye”) indi- cates an acceptable fit between the model and the data, which is confirmed by a χ2 goodness-of-fit test. This demonstrates that certain words seem to be adequately modeled by a Poisson distribution, whose probability mass function is shown in (1): Poisson(λ)(x) = λ x …

Content truncated to 500 words