Anchored Correlation Explanation: Topic Modeling with Minimal Domain Knowledge

Anchored Correlation Explanation: Topic Modeling with Minimal Domain Knowledge Ryan J. Gallagher1,2, Kyle Reing1, David Kale1, and Greg Ver Steeg1 1Information Sciences Institute, University of Southern California 2Vermont Complex Systems Center, Computational Story Lab, University of Vermont ryan.gallagher@uvm.edu {reing,kale,gregv}@isi.edu Abstract While generative models such as Latent Dirichlet Allocation (LDA) have proven fruit- ful in topic modeling, they often require de- tailed assumptions and careful specification of hyperparameters. Such model complexity is- sues only compound when trying to general- ize generative models to incorporate human input. We introduce Correlation Explanation (CorEx), an alternative approach to topic mod- eling that does not assume an underlying gen- erative model, and instead learns maximally informative topics through an information- theoretic framework. This framework nat- urally generalizes to hierarchical and semi- supervised extensions with no additional mod- eling assumptions. In particular, word-level domain knowledge can be flexibly incorpo- rated within CorEx through anchor words, al- lowing topic separability and representation to be promoted with minimal human interven- tion. Across a variety of datasets, metrics, and experiments, we demonstrate that CorEx produces topics that are comparable in quality to those produced by unsupervised and semi- supervised variants of LDA. 1 Introduction The majority of topic modeling approaches utilize probabilistic generative models, models which spec- ify mechanisms for how documents are written in order to infer latent topics. These mechanisms may be explicitly stated, as in Latent Dirichlet Alloca- tion (LDA) (Blei et al., 2003), or implicitly stated, as with matrix factorization techniques (Hofmann, 1999; Ding et al., 2008; Buntine and Jakulin, 2006). The core generative mechanisms of LDA, in par- ticular, have inspired numerous generalizations that account for additional information, such as the au- thorship (Rosen-Zvi et al., 2004), document labels (McAuliffe and Blei, 2008), or hierarchical structure (Griffiths et al., 2004). However, these generalizations come at the cost of increasingly elaborate and unwieldy generative assumptions. While these assumptions allow topic inference to be tractable in the face of additional metadata, they progressively constrain topics to a narrower view of what a topic can be. Such assump- tions are undesirable in contexts where one wishes to minimize model complexity and learn topics with- out preexisting notions of how those topics origi- nated. For these reasons, we propose topic modeling by way of Correlation Explanation (CorEx),1 an information-theoretic approach to learning latent topics over documents. Unlike LDA, CorEx does not assume a particular data generating model, and instead searches for topics that are “maximally in- formative” about a set of documents. By learning informative topics rather than generated topics, we avoid specifying the structure and nature of topics ahead of time. In addition, the lightweight framework underly- ing CorEx is versatile and naturally extends to hier- archical and semi-supervised variants with no addi- tional modeling assumptions. More specifically, we 1Open source, documented code for the CorEx topic model available at https://github.com/gregversteeg/ corex_topic. 529 Transactions of the Association for Computational Linguistics, vol. 5, pp. 529–542, 2017. Action Editor: Diana McCarthy. Submission batch: 7/2017 Published 12/2017. c⃝2017 Association for …

Content truncated to 500 words