Sprite: Generalizing Topic Models with Structured Priors

SPRITE: Generalizing Topic Models with Structured Priors Michael J. Paul and Mark Dredze Department of Computer Science Human Language Technology Center of Excellence Johns Hopkins University, Baltimore, MD 21218 mpaul@cs.jhu.edu, mdredze@cs.jhu.edu Abstract We introduce SPRITE, a family of topic models that incorporates structure into model priors as a function of underlying components. The structured priors can be constrained to model topic hierarchies, factorizations, correlations, and supervi- sion, allowing SPRITE to be tailored to particular settings. We demonstrate this flexibility by constructing a SPRITE-based model to jointly infer topic hierarchies and author perspective, which we apply to cor- pora of political debates and online re- views. We show that the model learns in- tuitive topics, outperforming several other topic models at predictive tasks. 1 Introduction Topic models can be a powerful aid for analyzing large collections of text by uncovering latent in- terpretable structures without manual supervision. Yet people often have expectations about topics in a given corpus and how they should be structured for a particular task. It is crucial for the user expe- rience that topics meet these expectations (Mimno et al., 2011; Talley et al., 2011) yet black box topic models provide no control over the desired output. This paper presents SPRITE, a family of topic models that provide a flexible framework for en- coding preferences as priors for how topics should be structured. SPRITE can incorporate many types of structure that have been considered in prior work, including hierarchies (Blei et al., 2003a; Mimno et al., 2007), factorizations (Paul and Dredze, 2012; Eisenstein et al., 2011), sparsity (Wang and Blei, 2009; Balasubramanyan and Co- hen, 2013), correlations between topics (Blei and Lafferty, 2007; Li and McCallum, 2006), pref- erences over word choices (Andrzejewski et al., 2009; Paul and Dredze, 2013), and associations between topics and document attributes (Ramage et al., 2009; Mimno and McCallum, 2008). SPRITE builds on a standard topic model, adding structure to the priors over the model pa- rameters. The priors are given by log-linear func- tions of underlying components (§2), which pro- vide additional latent structure that we will show can enrich the model in many ways. By apply- ing particular constraints and priors to the compo- nent hyperparameters, a variety of structures can be induced such as hierarchies and factorizations (§3), and we will show that this framework cap- tures many existing topic models (§4). After describing the general form of the model, we show how SPRITE can be tailored to partic- ular settings by describing a specific model for the applied task of jointly inferring topic hierar- chies and perspective (§6). We experiment with this topic+perspective model on sets of political debates and online reviews (§7), and demonstrate that SPRITE learns desired structures while outper- forming many baselines at predictive tasks. 2 Topic Modeling with Structured Priors Our model family generalizes latent Dirichlet al- location (LDA) (Blei et al., 2003b). Under LDA, there are K topics, where a topic is a categor- ical distribution over V words parameterized by φk. Each …

Content truncated to 500 words