A Bayesian Mixture Model for Term Re-occurrence and Burstiness

Proceedings of the 9th Conference on Computational Natural Language Learning (CoNLL), pages 48–55, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics A Bayesian mixture model for term re-occurrence and burstiness Avik Sarkar1, Paul H Garthwaite2, Anne De Roeck1 1 Department of Computing, 2 Department of Statistics The Open University Milton Keynes, MK7 6AA, UK {a.sarkar, p.h.garthwaite, a.deroeck}@open.ac.uk Abstract This paper proposes a model for term re- occurrence in a text collection based on the gaps between successive occurrences of a term. These gaps are modeled using a mixture of exponential distributions. Pa- rameter estimation is based on a Bayesian framework that allows us to fit a flexi- ble model. The model provides measures of a term’s re-occurrence rate and within- document burstiness. The model works for all kinds of terms, be it rare content word, medium frequency term or frequent function word. A measure is proposed to account for the term’s importance based on its distribution pattern in the corpus. 1 Introduction Traditionally, Information Retrieval (IR) and Statis- tical Natural Language Processing (NLP) applica- tions have been based on the “bag of words” model. This model assumes term independence and homo- geneity of the text and document under considera- tion, i.e. the terms in a document are all assumed to be distributed homogeneously. This immediately leads to the Vector Space representation of text. The immense popularity of this model is due to the ease with which mathematical and statistical techniques can be applied to it. The model assumes that once a term occurs in a document, its overall frequency in the entire doc- ument is the only useful measure that associates a term with a document. It does not take into consid- eration whether the term occurred in the beginning, middle or end of the document. Neither does it con- sider whether the term occurs many times in close succession or whether it occurs uniformly through- out the document. It also assumes that additional positional information does not provide any extra leverage to the performance of the NLP and IR ap- plications based on it. This assumption has been shown to be wrong in certain applications (Franz, 1997). Existing models for term distribution are based on the above assumption, so they can merely estimate the term’s frequency in a document or a term’s top- ical behavior for a content term. The occurrence of a content word is classified as topical or non-topical based on whether it occurs once or many times in the document (Katz, 1996). We are not aware of any existing model that makes less stringent assumptions and models the distribution of occurrences of a term. In this paper we describe a model for term re- occurrence in text based on the gaps between succes- sive occurrences of the term and the position of its first occurrence in a document. The gaps are mod- eled by a mixture of exponential distributions. Non- occurrence of a term in a document is modeled by the statistical concept of censoring, …

Content truncated to 500 words