Semantic Topic Models: Combining Word Distributional Statistics and Dictionary Definitions

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 552–561, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Semantic Topic Models: Combining Word Distributional Statistics and Dictionary Definitions Weiwei Guo Department of Computer Science, Columbia University, weiwei@cs.columbia.edu Mona Diab Center for Computational Learning Systems, Columbia University, mdiab@ccls.columbia.edu Abstract In this paper, we propose a novel topic model based on incorporating dictionary definitions. Traditional topic models treat words as surface strings without assuming predefined knowledge about word mean- ing. They infer topics only by observing surface word co-occurrence. However, the co-occurred words may not be semanti- cally related in a manner that is relevant for topic coherence. Exploiting dictionary definitions explicitly in our model yields a better understanding of word semantics leading to better text modeling. We exploit WordNet as a lexical resource for sense definitions. We show that explicitly mod- eling word definitions helps improve per- formance significantly over the baseline for a text categorization task. 1 Introduction Latent Dirichlet Allocation (LDA) (Blei et al., 2003) serves as a data-driven framework in model- ing text corpora. The statistical model allows vari- able extensions to integrate linguistic features such as syntax (Griffiths et al., 2005), and has been ap- plied in many areas. In LDA, there are two factors which determine the topic of a word: the topic distribution of the document, and the probability of a topic to emit this word. This information is learned in an unsu- pervised manner to maximize the likelihood of the corpus. However, this data-driven approach has some limitations. If a word is not observed fre- quently enough in the corpus, then it is likely to be assigned the dominant topic in this document. For example, the word grease (a thick fatty oil) in a political domain document should be assigned the topic chemicals. However, since it is an in- frequent word, LDA cannot learn its correct se- mantics from the observed distribution, the LDA model will assign it the dominant document topic politics. If we look up the semantics of the word grease in a dictionary, we will not find any of its meanings indicating the politics topic, yet there is ample evidence for the chemical topic. Accord- ingly, we hypothesize that if we know the seman- tics of words in advance, we can get a better in- dication of their topics. Therefore, in this paper, we test our hypothesis by exploring the integration of word semantics explicitly in the topic modeling framework. In order to incorporate word semantics from dictionaries, we recognize the need to model sense-topic distribution rather than word-topic dis- tribution, since dictionaries are constructed at the sense level. We use WordNet (Fellbaum, 1998) as our lexical resource of choice. The notion of a sense in WordNet goes beyond a typical word sense in a traditional dictionary since a WordNet sense links senses of different words that have similar meanings. Accordingly, the sense for the first verbal entry for buy and for purchase will …

Content truncated to 500 words