A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings

A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings Lixing Zhu† Yulan He†∗ Deyu Zhou§ †Department of Computer Science, University of Warwick, UK §School of Computer Science and Engineering, Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, China {lixing.zhu, yulan.he}@warwick.ac.uk d.zhou@seu.edu.cn Abstract We propose a novel generative model to ex- plore both local and global context for joint learning topics and topic-specific word embed- dings. In particular, we assume that global latent topics are shared across documents, a word is generated by a hidden semantic vector encoding its contextual semantic meaning, and its context words are generated conditional on both the hidden semantic vector and global latent topics. Topics are trained jointly with the word embeddings. The trained model maps words to topic-dependent embeddings, which naturally addresses the issue of word poly- semy. Experimental results show that the proposed model outperforms the word-level embedding methods in both word similarity evaluation and word sense disambiguation. Furthermore, the model also extracts more coherent topics compared with existing neural topic models or other models for joint learning of topics and word embeddings. Finally, the model can be easily integrated with existing deep contextualized word embedding learning methods to further improve the performance of downstream tasks such as sentiment classification. 1 Introduction Probabilistic topic models assume that words are generated from latent topics that can be inferred from word co-occurrence patterns taking a docu- ment as global context. In recent years, various neural topic models have been proposed. Some of them are built on the Variational Auto-Encoder (VAE) (Kingma and Welling, 2014), which uti- lizes deep neural networks to approximate the in- ∗Corresponding author. tractable posterior distribution of observed words given latent topics (Miao et al., 2016; Srivastava and Sutton, 2017; Bouchacourt et al., 2018). However, these models take the bag-of-words (BOWs) representation of a given document as the input to the VAE and aim to learn hidden topics that can be used to reconstruct the original document. They do not learn word embeddings concurrently. Other topic modeling approaches explore the pre-trained word embeddings for the extraction of more semantically coherent topics since word embeddings capture syntactic and semantic regu- larities by encoding the local context of word co-occurrence patterns. For example, the topic- word generation process in the traditional topic models can be replaced by generating word embeddings given latent topics (Das et al., 2015) or by a two-component mixture of a Dirichlet multinomial component and a word embedding component (Nguyen et al., 2015). Alternatively, the information derived from word embeddings can be used to promote semantically related words in the Polya Urn sampling process of topic models (Li et al., 2017) or generate topic hierarchies (Zhao et al., 2018). However, all these models use pre- trained word embeddings and do not learn word embeddings jointly with topics. Word embeddings could improve the topic modeling results, butconversely, the topic informa- tion could also benefit word embedding learning. Earlyword embedding learning methods (Mikolov et al., 2013a) …

Content truncated to 500 words