Deep Dirichlet Multinomial Regression

Proceedings of NAACL-HLT 2018, pages 365–374 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Deep Dirichlet Multinomial Regression Adrian Benton Mark Dredze Center for Language and Speech Processing Johns Hopkins University Baltimore, MD 21218, USA {adrian,mdredze}@cs.jhu.edu Abstract Dirichlet Multinomial Regression (DMR) and other supervised topic models can in- corporate arbitrary document-level features to inform topic priors. However, their abil- ity to model corpora are limited by the rep- resentation and selection of these features – a choice the topic modeler must make. In- stead, we seek models that can learn the feature representations upon which to con- dition topic selection. We present deep Dirichlet Multinomial Regression (dDMR), a generative topic model that simultane- ously learns document feature representa- tions and topics. We evaluate dDMR on three datasets: New York Times articles with fine-grained tags, Amazon product reviews with product images, and Reddit posts with subreddit identity. dDMR learns representations that outperform DMR and LDA according to heldout perplexity and are more effective at downstream predic- tive tasks as the number of topics grows. Additionally, human subjects judge dDMR topics as being more representative of as- sociated document features. Finally, we find that supervision leads to faster conver- gence as compared to an LDA baseline and that dDMR’s model fit is less sensitive to training parameters than DMR. 1 Introduction Fifteen years of research on topic models, starting from Latent Dirichlet Allocation (LDA) (Blei et al., 2003), have led to a variety of models for numerous data settings. These models identify sets (distribu- tions) of related words that reflect semantic topics in a large corpus of text data. Topic models are now routinely used in the social sciences and humanities to analyze text collections (Schmidt, 2012). Document collections are often accompanied by metadata and annotations, such as a book’s author, an article’s topic descriptor tags, images associated with a product review, or structured patient in- formation associated with clinical records. These document-level annotations can provide additional supervision for guiding topic model learning. Ad- ditional information can be integrated into topic models using either downstream or upstream mod- els. Downstream models, such as supervised LDA (Mcauliffe and Blei, 2008), assume that these addi- tional document features are generated from each document’s topic distribution. These models are most helpful when you desire topics that are pre- dictive of the output, such as models for predict- ing the sentiment of product reviews. Upstream models, such as Dirichlet Multinomial Regression (DMR), condition each document’s topic distribu- tion on document features, such as author (Rosen- Zvi et al., 2004), social network (McCallum et al., 2007), or document labels (Ramage et al., 2009). Previous work has demonstrated that upstream models tend to outperform downstream models in terms of model fit, as well as extracting topics that are useful in prediction of related tasks (Benton et al., 2016). DMR is an upstream topic model with a particu- larly attractive method for incorporating arbitrary document features. Rather than defining specific random variables in the …

Content truncated to 500 words