Latent Topic Embedding

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2689–2698, Osaka, Japan, December 11-17 2016. Latent Topic Embedding Di Jiang Baidu, Inc., China jiangdi@baidu.com Rongzhong Lian Baidu, Inc., China lianrongzhong@baidu.com Lei Shi Baidu, Inc., China shilei06@baidu.com Hua Wu Baidu, Inc., China wu hua@baidu.com Abstract Topic modeling and word embedding are two important techniques for deriving latent semantics from data. General-purpose topic models typically work in coarse granularity by capturing word co-occurrence at the document/sentence level. In contrast, word embedding models usually work in fine granularity by modeling word co-occurrence within small sliding windows. With the aim of deriving latent semantics by capturing word co-occurrence information at different levels of granularity, we propose a novel model named Latent Topic Embedding (LTE), which seamlessly integrates topic generation and embedding learning in one unified framework. We further propose an efficient Monte Carlo EM algorithm to estimate the parameters of interest. By retaining the individual advantages of topic modeling and word embedding, LTE results in better latent topics and word embedding. Experimental results verify the superiority of LTE over the state-of-the-arts in real-life applications. 1 Introduction Topic modeling and word embedding are gaining significant momentum in the field of text mining. General-purpose topic models such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003) and Sentence LDA (Jo and Oh, 2011) usually utilize word co-occurrences at the document/sentence level to compose the ”topics”, which capture the latent semantics between words. These models are plagued by the sim- plistic bag-of-words assumption, which ignores the valuable sub-sequence information between words. Some recent endeavors introduced n-gram information into topic models (Wallach, 2006), however, the size of vocabulary is significantly enlarged and these techniques are hardly feasible for real-life applica- tions. Therefore, the technique of topic modeling needs a remedy for solving the word sequence problem with fairly low cost. Word embedding models such as Word2Vec (Mikolov et al., 2013a) map words into distributed representations. Word embedding models primarily focus on the word co-occurrences within small sliding windows, which enable word embedding to capture (at least partially) the information of word sequences. One key problem of the existing word embedding models is that they are typically short-sighted and are not aware of the themes of the document. With their differences, the core of topic modeling and word embedding is based upon the assumption that the words co-occurring frequently should have semantic commonality. In light of their individual advantages and drawbacks, we see that the two techniques are essentially complimentary and can be inte- grated to enhance each other. In this paper, we propose the Latent Topic Embedding (LTE) to seamlessly integrate topic modeling and word embedding in one framework. In the generative process of LTE, we assume that the observed words in document can be generated through two channels: one is through the Multinomial distribution and the other is based upon topic embeddings as well as word embeddings. In this way, the embedding information influences the result of topic modeling while the topic …

Content truncated to 500 words