Improving Topic Coherence with Latent Feature Word Representations in MAP Estimation for Topic Modeling

Improving Topic Coherence with Latent Feature Word Representations in MAP Estimation for Topic Modeling Dat Quoc Nguyen, Kairit Sirts and Mark Johnson Department of Computing Macquarie University, Australia dat.nguyen@students.mq.edu.au, {kairit.sirts, mark.johnson}@mq.edu.au Abstract Probabilistic topic models are widely used to discover latent topics in document col- lections, while latent feature word vec- tors have been used to obtain high per- formance in many natural language pro- cessing (NLP) tasks. In this paper, we present a new approach by incorporating word vectors to directly optimize the max- imum a posteriori (MAP) estimation in a topic model. Preliminary results show that the word vectors induced from the experi- mental corpus can be used to improve the assignments of topics to words. Keywords: MAP estimation, LDA, Topic model, Word vectors, Topic coherence 1 Introduction Topic modeling algorithms, such as Latent Dirich- let Allocation (LDA) (Blei et al., 2003) and re- lated methods (Blei, 2012), are often used to learn a set of latent topics for a corpus of docu- ments and to infer document-to-topic and topic- to-word distributions from the co-occurrence of words within the documents (Wallach, 2006; Blei and McAuliffe, 2008; Wang et al., 2007; Johnson, 2010; Yan et al., 2013; Xie et al., 2015; Yang et al., 2015). With enough training data there is suf- ficient information in the corpus to accurately esti- mate the distributions. However, most topic mod- els consider each document as a bag-of-words, i.e. the word order or the window-based local context information is not taken into account. Topic models have also been constructed using latent features (Salakhutdinov and Hinton, 2009; Srivastava et al., 2013; Cao et al., 2015). La- tent feature vectors have been recently success- fully exploited for a wide range of NLP tasks (Glorot et al., 2011; Socher et al., 2013; Pen- nington et al., 2014). Rather than relying solely on word count information as the standard multi- nomial LDA does, or using only distributed fea- ture representations, as in Salakhutdinov and Hin- ton (2009), Srivastava et al. (2013) and Cao et al. (2015), Nguyen et al. (2015) integrated pre- trained latent feature word representations con- taining external information from very large cor- pora into existing topic models and obtained sig- nificant improvements on small document collec- tions and short text datasets. However, their im- plementation is computationally quite expensive because they have to compute a MAP estimate in each Gibbs sampling iteration. In this paper, we experiment with MAP estima- tion using word vectors for LDA. Instead of mixing the Gibbs sampling and MAP estimation, we pro- pose to optimize the MAP estimation of the full model directly. In addition, instead of using the pre-trained word vectors learned on external large corpora, we propose to learn the internal word vec- tors from the same topic-modeling corpus that we induce the document-to-topic and topic-to-word distributions from. In this manner, we can also handle the words that are not found in the list of the pre-trained word vectors. Furthermore, the in- ternal word vectors can capture …

Content truncated to 500 words