Improving Topic Models with Latent Feature Word Representations

Improving Topic Models with Latent Feature Word Representations Dat Quoc Nguyen1, Richard Billingsley1, Lan Du1 and Mark Johnson1,2 1 Department of Computing, Macquarie University, Sydney, Australia 2 Santa Fe Institute, Santa Fe, New Mexico, USA dat.nguyen@students.mq.edu.au, {richard.billingsley, lan.du, mark.johnson}@mq.edu.au Abstract Probabilistic topic models are widely used to discover latent topics in document collec- tions, while latent feature vector representa- tions of words have been used to obtain high performance in many NLP tasks. In this pa- per, we extend two different Dirichlet multino- mial topic models by incorporating latent fea- ture vector representations of words trained on very large corpora to improve the word-topic mapping learnt on a smaller corpus. Exper- imental results show that by using informa- tion from the external corpora, our new mod- els produce significant improvements on topic coherence, document clustering and document classification tasks, especially on datasets with few or short documents. 1 Introduction Topic modeling algorithms, such as Latent Dirichlet Allocation (Blei et al., 2003) and related methods (Blei, 2012), are often used to learn a set of latent topics for a corpus, and predict the probabilities of each word in each document belonging to each topic (Teh et al., 2006; Newman et al., 2006; Toutanova and Johnson, 2008; Porteous et al., 2008; Johnson, 2010; Xie and Xing, 2013; Hingmire et al., 2013). Conventional topic modeling algorithms such as these infer document-to-topic and topic-to-word dis- tributions from the co-occurrence of words within documents. But when the training corpus of docu- ments is small or when the documents are short, the resulting distributions might be based on little evi- dence. Sahami and Heilman (2006) and Phan et al. (2011) show that it helps to exploit external knowl- edge to improve the topic representations. Sahami and Heilman (2006) employed web search results to improve the information in short texts. Phan et al. (2011) assumed that the small corpus is a sample of topics from a larger corpus like Wikipedia, and then use the topics discovered in the larger corpus to help shape the topic representations in the small corpus. However, if the larger corpus has many irrel- evant topics, this will “use up” the topic space of the model. In addition, Petterson et al. (2010) proposed an extension of LDA that uses external information about word similarity, such as thesauri and dictio- naries, to smooth the topic-to-word distribution. Topic models have also been constructed using la- tent features (Salakhutdinov and Hinton, 2009; Sri- vastava et al., 2013; Cao et al., 2015). Latent fea- ture (LF) vectors have been used for a wide range of NLP tasks (Glorot et al., 2011; Socher et al., 2013; Pennington et al., 2014). The combination of val- ues permitted by latent features forms a high dimen- sional space which makes it is well suited to model topics of very large corpora. Rather than relying solely on a multinomial or la- tent feature model, as in Salakhutdinov and Hinton (2009), Srivastava et al. (2013) and Cao et al. (2015), we explore how …

Content truncated to 500 words