Integration of Knowledge Graph Embedding Into Topic Modeling with Hierarchical Dirichlet Process

Proceedings of NAACL-HLT 2019, pages 940–950 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 940 Integration of Knowledge Graph Embedding into Topic Modeling with Hierarchical Dirichlet Process Dingcheng Li, Siamak Zamani Dadaneh, Jingyuan Zhang, Ping Li Cognitive Computing Lab (CCL) Baidu Research USA 10900 NE Eighth St, Bellevue, WA 98004, USA 1195 Bordeaux Dr, Sunnyvale, CA 94089, USA lidingcheng@baidu.com, zamanys4@gmail.com zhangjingyuan03@baidu.com, liping11@baidu.com Abstract Leveraging domain knowledge is an effective strategy for enhancing the quality of inferred low-dimensional representations of documents by topic models. In this paper, we develop topic modeling with knowledge graph em- bedding (TMKGE), a Bayesian nonparamet- ric model to employ knowledge graph (KG) embedding in the context of topic modeling, for extracting more coherent topics. Specifi- cally, we build a hierarchical Dirichlet process (HDP) based model to flexibly borrow infor- mation from KG to improve the interpretabil- ity of topics. An efficient online variational in- ference method based on a stick-breaking con- struction of HDP is developed for TMKGE, making TMKGE suitable for large document corpora and KGs. Experiments on three pub- lic datasets illustrate the superior performance of TMKGE in terms of topic coherence and document classification accuracy, compared to state-of-the-art topic modeling methods. 1 Introduction Topic models, such as Probabilistic Latent Seman- tic Analysis (PLSA) (Hofmann, 2017) and La- tent Dirichlet Allocation (LDA) (Blei et al., 2003), play significant roles in helping machines inter- pret text documents. Topic models consider doc- uments as a bag of words. Given the word in- formation, topic models formulate documents as mixtures of latent topics, where these topics are generated via the multinomial distributions over words. Bayesian methods are utilized to extract topical structures from the document-word fre- quency representations of the text corpus. With- out supervision, however, it is found that the top- ics generated from these models are often not interpretable (Chang et al., 2009; Mimno et al., 2011). In recent studies, incorporating knowledge of different forms as a supervision has become a powerful strategy for discovering meaningful top- ics (Andrzejewski et al., 2009). president male born wife Barack Obama United States Honolulu Michelle Obama wasBornIn hasGender livesIn locatedIn Word Co-occurrence Entity Embeddings Relation Embeddings Latent Topics Words Entities Relations Documents Knowledge Graphs Online stochastic Optimization with Natural gradients for varitional inferences Figure 1: An overview of the proposed TMKGE frame- work. Entities are shared by both documents and knowledge graphs. Entity embeddings generated by TransE a knowledge graph embedding package are passed into TMKGE to generate hidden topics. Most conventional approaches take prior do- main knowledge into account to improve the topic coherence (Andrzejewski et al., 2009; Andrze- jewski and Zhu, 2009; Hu et al., 2014; Jagar- lamudi et al., 2012; Doshi-Velez et al., 2015). One commonly used domain knowledge is based on word correlations (Andrzejewski et al., 2009; Chen et al., 2013; Chen and Liu, 2014). For exam- ple, must-links and cannot-links among words are generated by domain experts to help topic mod- eling (Andrzejewski et al., 2009). Another useful …

Content truncated to 500 words