An Automatic Approach for Document-level Topic Model Evaluation

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 206–215, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics An Automatic Approach for Document-level Topic Model Evaluation Shraey Bhatia1 Jey Han Lau1,2 Timothy Baldwin1 1 School of Computing and Information Systems, The University of Melbourne 2 IBM Research shraeybhatia@gmail.com, jeyhan.lau@gmail.com, tb@ldwin.net Abstract Topic models jointly learn topics and document-level topic distribution. Extrin- sic evaluation of topic models tends to fo- cus exclusively on topic-level evaluation, e.g. by assessing the coherence of topics. We demonstrate that there can be large dis- crepancies between topic- and document- level model quality, and that basing model evaluation on topic-level analysis can be highly misleading. We propose a method for automatically predicting topic model quality based on analysis of document- level topic allocations, and provide empiri- cal evidence for its robustness. 1 Introduction Topic models such as latent Dirichlet allocation (Blei et al., 2003) jointly learn latent topics (in the form of multinomial distributions over words) and topic allocations to individual documents (in the form of multinomial distributions over top- ics), and provide a powerful means of document collection navigation and visualisation (Newman et al., 2010a; Chaney and Blei, 2012; Smith et al., 2017). One property of LDA-style topic models that has contributed to their popularity is that they are highly configurable, and can be structured to capture a myriad of statistical dependencies, such as between topics (Blei and Lafferty, 2006), be- tween documents associated with the same individ- ual (Rosen-Zvi et al., 2004), or between documents associated with individuals in different network relations (Wang and Blei, 2011). This has led to a wealth of topic models of different types, and the need for methods to evaluate different styles of topic model over the same document collections. Test data perplexity is the obvious solution, but it has been shown to correlate poorly with direct human assessment of topic model quality (Chang et al., 2009), motivating the need for automatic topic model evaluation methods which emulate hu- man assessment. Research in this vein has focused primarily on evaluating the quality of individual topics (Newman et al., 2010b; Mimno et al., 2011; Aletras and Stevenson, 2013; Lau et al., 2014; Fang et al., 2016) and largely ignored evaluation of topic allocations to individual documents, and it has be- come widely accepted that topic-level evaluation is a reliable indicator of the intrinsic quality of the overall topic model (Lau et al., 2014). We chal- lenge this assumption, and demonstrate that topic model evaluation should operate at both the topic and document levels. Our primary contributions are as follows: (1) we empirically demonstrate that there can be large discrepancies between topic- and document-level topic model evaluation; (2) we demonstrate that previously-proposed document-level evaluation ap- proaches can be misleading, and propose an al- ternative evaluation method; and (3) we propose an automatic approach to topic model evaluation based on analysis of document-level topic distri- butions, which we show to correlate strongly with …

Content truncated to 500 words