Authorless Topic Models: Biasing Models Away from Known Structure

Proceedings of the 27th International Conference on Computational Linguistics, pages 3903–3914 Santa Fe, New Mexico, USA, August 20-26, 2018. 3903 Authorless Topic Models: Biasing Models Away from Known Structure Laure Thompson Department of Computer Science Cornell University Ithaca, NY 14853 laurejt@cs.cornell.edu David Mimno Department of Information Science Cornell University Ithaca, NY 14853 mimno@cornell.edu Abstract Most previous work in unsupervised semantic modeling in the presence of metadata has assumed that our goal is to make latent dimensions more correlated with metadata, but in practice the exact opposite is often true. Some users want topic models that highlight differences between, for example, authors, but others seek more subtle connections across authors. We introduce three metrics for identifying topics that are highly correlated with metadata, and demonstrate that this problem affects between 30 and 50% of the topics in models trained on two real-world collections, regardless of the size of the model. We find that we can predict which words cause this phenomenon and that by selectively subsampling these words we dramatically reduce topic- metadata correlation, improve topic stability, and maintain or even improve model quality. 1 Introduction Unsupervised semantic models are a popular and useful method for inferring low-dimensional representa- tions of large text collections. Examples of such models include latent semantic analysis (Deerwester et al., 1990) and word embeddings (Bengio et al., 2003), but for this work we will focus on statistical topic models (Hofmann, 1999; Blei et al., 2002), which are used to infer word distributions that correspond to recognizable themes. In practice, collections are often constructed by combining documents from multiple sources, which may have distinctive style and vocabulary. This heterogeneity of sources leads to a serious but rarely studied problem: the strongest, most prominent patterns in a collection may simply repeat the known structure of the corpus. Instead of finding informative, cross-cutting themes, models simply repeat the distinctive vocabulary of the individual sources. The model in this case is “correct” in that it has detected the strongest dimensions of variation, but it tells us nothing we did not already know. As a motivating example, we focus on models trained on novels, where it is known that inferred topics are often simply names of characters and settings (Jockers, 2013). The words Harry, Ron, and Hermione look to the algorithm like the basis of an ideal topic because they occur very frequently together but not in other contexts. But this topic only tells us which books within a larger corpus are part of the Harry Potter series; themes like friendship, adolescence, and magic remain hidden. This phenomenon is not limited to fiction: we also include a case study of opinions from US state supreme courts. Unlike examples from fiction, Maine and Utah both exist in the same universe, but exhibit specific regional term use. We begin by demonstrating that the problem of overly source-specific topics is both substantial and measurable. We present three metrics that provide related but distinct views of source specificity. These metrics are orthogonal to existing metrics …

Content truncated to 500 words