Term Weighting Schemes for Latent Dirichlet Allocation

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 465–473, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Term Weighting Schemes for Latent Dirichlet Allocation Andrew T. Wilson Sandia National Laboratories PO Box 5800, MS 1323 Albuquerque, NM 87185-1323, USA atwilso@sandia.gov Peter A. Chew Moss Adams LLP 6100 Uptown Blvd. NE, Suite 400 Albuquerque, NM 87110-4489, USA Peter.Chew@MossAdams.com Abstract Many implementations of Latent Dirichlet Al- location (LDA), including those described in Blei et al. (2003), rely at some point on the removal of stopwords, words which are as- sumed to contribute little to the meaning of the text. This step is considered necessary be- cause otherwise high-frequency words tend to end up scattered across many of the latent top- ics without much rhyme or reason. We show, however, that the ‘problem’ of high-frequency words can be dealt with more elegantly, and in a way that to our knowledge has not been considered in LDA, through the use of appro- priate weighting schemes comparable to those sometimes used in Latent Semantic Indexing (LSI). Our proposed weighting methods not only make theoretical sense, but can also be shown to improve precision significantly on a non-trivial cross-language retrieval task. 1 Introduction Latent Dirichlet Allocation (LDA) (Blei et al., 2003), like its more established competitors Latent Seman- tic Indexing (LSI) (Deerwester et al., 1990) and Probabilistic Latent Semantic Indexing (PLSI) (Hof- mann, 1999), is a model which is applicable to the analysis of text corpora. It is claimed to differ from LSI in that LDA is a generative Bayesian model (Blei et al., 2003), although this may depend upon the manner in which one approaches LSI (see for exam- ple Chew et al. (2010)). In LDA as applied to text analysis, each document in the corpus is modeled as a mixture over an underlying set of topics, and each topic is modeled as a probability distribution over the terms in the vocabulary. As the newest among the above-mentioned tech- niques, LDA is still in a relatively early stage of de- velopment. It is also sufficiently different from LSI, probably the most popular and well-known compres- sion technique for information retrieval (IR), that many practitioners of LSI may perceive a ‘barrier to entry’ to LDA. This in turn perhaps explains why no- tions such as term weighting, which have been com- monplace in LSI for some time (Dumais, 1991), have not yet found a place in LDA. In fact, it is often as- sumed that weighting is unnecessary in LDA. For example, Blei et al. (2003) contrast the use of tf- idf weighting in both non-reduced space (Salton and McGill, 1983) and LSI on the one hand with PLSI and LDA on the other, where no mention is made of weighting. Ramage et al. (2008) propose a simple term-frequency weighting scheme for tagged docu- ments within the framework of LDA, although term weighting is not their focus and their scheme is in- tended to incorporate document …

Content truncated to 500 words