Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 248–256, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora Daniel Ramage, David Hall, Ramesh Nallapati and Christopher D. Manning Computer Science Department Stanford University {dramage,dlwh,nmramesh,manning}@cs.stanford.edu Abstract A significant portion of the world’s text is tagged by readers on social bookmark- ing websites. Credit attribution is an in- herent problem in these corpora because most pages have multiple tags, but the tags do not always apply with equal specificity across the whole document. Solving the credit attribution problem requires associ- ating each word in a document with the most appropriate tags and vice versa. This paper introduces Labeled LDA, a topic model that constrains Latent Dirichlet Al- location by defining a one-to-one corre- spondence between LDA’s latent topics and user tags. This allows Labeled LDA to directly learn word-tag correspondences. We demonstrate Labeled LDA’s improved expressiveness over traditional LDA with visualizations of a corpus of tagged web pages from del.icio.us. Labeled LDA out- performs SVMs by more than 3 to 1 when extracting tag-specific document snippets. As a multi-label text classifier, our model is competitive with a discriminative base- line on a variety of datasets. 1 Introduction From news sources such as Reuters to modern community web portals like del.icio.us, a signif- icant proportion of the world’s textual data is la- beled with multiple human-provided tags. These collections reflect the fact that documents are often about more than one thing—for example, a news story about a highway transportation bill might naturally be filed under both transportation and politics, with neither category acting as a clear subset of the other. Similarly, a single web page in del.icio.us might well be annotated with tags as diverse as arts, physics, alaska, and beauty. However, not all tags apply with equal speci- ficity across the whole document, opening up new opportunities for information retrieval and cor- pus analysis on tagged corpora. For instance, users who browse for documents with a particu- lar tag might prefer to see summaries that focus on the portion of the document most relevant to the tag, a task we call tag-specific snippet extrac- tion. And when a user browses to a particular document, a tag-augmented user interface might provide overview visualization cues highlighting which portions of the document are more or less relevant to the tag, helping the user quickly access the information they seek. One simple approach to these challenges can be found in models that explicitly address the credit attribution problem by associating individ- ual words in a document with their most appropri- ate labels. For instance, in our news story about the transportation bill, if the model knew that the word “highway” went with transportation and that the word “politicians” went with politics, more relevant passages could be extracted for either la- bel. We seek an approach that can automatically learn the posterior distribution of each word in a document conditioned on …

Content truncated to 500 words