Adapting Topic Models using Lexical Associations with Tree Priors

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1901–1906 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Adapting Topic Models using Lexical Associations with Tree Priors Weiwei Yang Computer Science (CS) University of Maryland College Park, MD wwyang@cs.umd.edu Jordan Boyd-Graber CS, iSchool, LSC, and UMIACS University of Maryland College Park, MD jbg@umiacs.umd.edu Philip Resnik Linguistics and UMIACS University of Maryland College Park, MD resnik@umd.edu Abstract Models work best when they are opti- mized taking into account the evaluation criteria that people care about. For topic models, people often care about inter- pretability, which can be approximated us- ing measures of lexical association. We in- tegrate lexical association into topic opti- mization using tree priors, which provide a flexible framework that can take advan- tage of both first order word associations and the higher-order associations captured by word embeddings. Tree priors improve topic interpretability without hurting ex- trinsic performance. 1 Introduction Goodman (1996) introduces a key insight for ma- chine learning models in natural language process- ing: if you know how performance on a problem is evaluated, it makes more sense to optimize using that evaluation metric, rather than others. Good- man applies his insight to parsing algorithms, but this insight has had an even larger impact in ma- chine translation, where the introduction of the fully automatic BLEU metric makes it possible to tune systems using a score correlated with hu- man rankings of MT system performance (Pap- ineni et al., 2002). Chang et al. (2009) provide a similar insight for topic models (Blei et al., 2003, LDA): if what you care about is the interpretability of topics, the standard objective function for parameter in- ference (likelihood) is not only poorly correlated with a human-centered measurement of topic co- herence, but inversely correlated. Nonetheless, most topic models are still trained using meth- ods that optimize likelihood (McAuliffe and Blei, 2008; Nguyen et al., 2013). We take the logical next step suggested when you bring together the insights of Goodman (1996) and Chang et al. (2009), namely incorporating an approximation of human topic interpretabil- ity into the topic model optimization process in a way that is effective and more straightforward than previous methods (Newman et al., 2011). We take advantage of the human-centered evaluation of Chang et al. (2009), which can be reasonably approximated using an automatic metric based on word associations derived from a large, more gen- eral corpus (Lau et al., 2014). We exploit LDA and its Bayesian formulation by bringing word associ- ations into the picture using a prior—specifically, we use external lexical association to create a tree structure and then use tree LDA (Boyd-Graber et al., 2007, tLDA), which derives topics using a given tree prior. We construct tree priors with combinations of two types of word association scores (skip-gram probability (Mikolov et al., 2013) and G2 likeli- hood ratio (Dunning, 1993)) and three construc- tion algorithms (two-level, hierarchical clustering with and without leaf duplication). Then tLDA identifies topics with …

Content truncated to 500 words