Generative Topic Embedding: a Continuous Representation of Documents

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 666–675, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Generative Topic Embedding: a Continuous Representation of Documents Shaohua Li1,2 Tat-Seng Chua1 Jun Zhu3 Chunyan Miao2 shaohua@gmail.com dcscts@nus.edu.sg dcszj@tsinghua.edu.cn ascymiao@ntu.edu.sg 1. School of Computing, National University of Singapore 2. Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY) 3. Department of Computer Science and Technology, Tsinghua University Abstract Word embedding maps words into a low- dimensional continuous embedding space by exploiting the local word collocation patterns in a small context window. On the other hand, topic modeling maps docu- ments onto a low-dimensional topic space, by utilizing the global word collocation patterns in the same document. These two types of patterns are complementary. In this paper, we propose a generative topic embedding model to combine the two types of patterns. In our model, topics are represented by embedding vectors, and are shared across documents. The proba- bility of each word is influenced by both its local context and its topic. A variational inference method yields the topic embed- dings as well as the topic mixing propor- tions for each document. Jointly they rep- resent the document in a low-dimensional continuous space. In two document clas- sification tasks, our method performs bet- ter than eight existing methods, with fewer features. In addition, we illustrate with an example that our method can generate co- herent topics even based on only one doc- ument. 1 Introduction Representing documents as fixed-length feature vectors is important for many document process- ing algorithms. Traditionally documents are rep- resented as a bag-of-words (BOW) vectors. How- ever, this simple representation suffers from being high-dimensional and highly sparse, and loses se- mantic relatedness across the vector dimensions. Word Embedding methods have been demon- strated to be an effective way to represent words as continuous vectors in a low-dimensional em- bedding space (Bengio et al., 2003; Mikolov et al., 2013; Pennington et al., 2014; Levy et al., 2015). The learned embedding for a word encodes its semantic/syntactic relatedness with other words, by utilizing local word collocation patterns. In each method, one core component is the embed- ding link function, which predicts a word’s distri- bution given its context words, parameterized by their embeddings. When it comes to documents, we wish to find a method to encode their overall semantics. Given the embeddings of each word in a document, we can imagine the document as a “bag-of-vectors”. Related words in the document point in similar di- rections, forming semantic clusters. The centroid of a semantic cluster corresponds to the most rep- resentative embedding of this cluster of words, re- ferred to as the semantic centroids. We could use these semantic centroids and the number of words around them to represent a document. In addition, for a set of documents in a partic- ular domain, some semantic clusters may appear in many documents. By learning collocation pat- terns across the documents, the derived semantic …

Content truncated to 500 words