Incorporating Lexical Priors into Topic Models

Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 204–213, Avignon, France, April 23 - 27 2012. c⃝2012 Association for Computational Linguistics Incorporating Lexical Priors into Topic Models Jagadeesh Jagarlamudi University of Maryland College Park, USA jags@umiacs.umd.edu Hal Daum´e III University of Maryland College Park, USA hal@umiacs.umd.edu Raghavendra Udupa Microsoft Research Bangalore, India raghavu@microsoft.com Abstract Topic models have great potential for help- ing users understand document corpora. This potential is stymied by their purely un- supervised nature, which often leads to top- ics that are neither entirely meaningful nor effective in extrinsic tasks (Chang et al., 2009). We propose a simple and effective way to guide topic models to learn topics of specific interest to a user. We achieve this by providing sets of seed words that a user believes are representative of the un- derlying topics in a corpus. Our model uses these seeds to improve both topic- word distributions (by biasing topics to pro- duce appropriate seed words) and to im- prove document-topic distributions (by bi- asing documents to select topics related to the seed words they contain). Extrinsic evaluation on a document clustering task reveals a significant improvement when us- ing seed information, even over other mod- els that use seed information na¨ıvely. 1 Introduction Topic models such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003) have emerged as a pow- erful tool to analyze document collections in an unsupervised fashion. When fit to a document collection, topic models implicitly use document level co-occurrence information to group seman- tically related words into a single topic. Since the objective of these models is to maximize the prob- ability of the observed data, they have a tendency to explain only the most obvious and superficial aspects of a corpus. They effectively sacrifice per- formance on rare topics to do a better job in mod- eling frequently occurring words. The user is then left with a skewed impression of the corpus, and perhaps one that does not perform well in extrin- sic tasks. To illustrate this problem, we ran LDA on the most frequent five categories of the Reuters- 21578 (Lewis et al., 2004) text corpus. This doc- ument distribution is very skewed: more than half of the collection belongs to the most frequent cat- egory (“Earn”). The five topics identified by the LDA are shown in Table 1. A brief observation of the topics reveals that LDA has roughly allo- cated topics 1 & 2 for the most frequent class (“Earn”) and one topic for the subsequent two frequent classes (“Acquisition” and “Forex”) and merged the least two frequent classes (“Crude” and “Grain”) into a single topic. The red colored words in topic 5 correspond to the “Crude” class and blue words are from the “Grain” class. This leads to the situation where the topics identified by LDA are not in accordance with the underlying topical structure of the corpus. This is a problem not just with LDA: it is potentially …

Content truncated to 500 words