Multi-source Neural Topic Modeling in Multi-view Embedding Spaces

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4205–4217 June 6–11, 2021. ©2021 Association for Computational Linguistics 4205 Multi-source Neural Topic Modeling in Multi-view Embedding Spaces *Pankaj Gupta1, *Yatin Chaudhary1,2, Hinrich Schütze2 1DRIMCo GmbH Munich, Germany 2CIS, University of Munich (LMU) Munich, Germany info@drimco.net | yatin.chaudhary@drimco.net Abstract Though word embeddings and topics are com- plementary representations, several past works have only used pretrained word embeddings in (neural) topic modeling to address data spar- sity in short-text or small collection of docu- ments. This work presents a novel neural topic modeling framework using multi-view embed- ding spaces: (1) pretrained topic-embeddings, and (2) pretrained word-embeddings (context- insensitive from Glove and context-sensitive from BERT models) jointly from one or many sources to improve topic quality and bet- ter deal with polysemy. In doing so, we first build respective pools of pretrained topic (i.e., TopicPool) and word embeddings (i.e., WordPool). We then identify one or more relevant source domain(s) and transfer knowl- edge to guide meaningful learning in the sparse target domain. Within neural topic mod- eling, we quantify the quality of topics and document representations via generalization (perplexity), interpretability (topic coherence) and information retrieval (IR) using short-text, long-text, small and large document collec- tions from news and medical domains. In- troducing the multi-source multi-view embed- ding spaces, we have shown state-of-the-art neural topic modeling using 6 source (high- resource) and 5 target (low-resource) corpora. 1 Introduction Probabilistic topic models, such as LDA (Blei et al., 2003), Replicated Softmax (RSM) (Salakhutdi- nov and Hinton, 2009) and Document Neural Au- toregressive Distribution Estimator (DocNADE) (Larochelle and Lauly, 2012) are often used to extract topics from text collections and learn la- tent document representations to perform natural language processing tasks, such as information re- trieval (IR). Though they have been shown to be powerful in modeling large text corpora, the topic * : equal contribution Topic Topic Words Topic Label Z1 (S1) profit, growth, stocks, apple, fall, Trading consumer, buy, billion, shares Z2(S2) smartphone, ipad, apple, app, Product Line iphone, devices, phone, tablet Z3 (S3) microsoft, mac, linux, ibm, ios, Operating System apple, xp, windows, software Z4 (T ) apple, talk, computers, shares, ? disease, driver, electronics, profit, ios Table 1: Coherent (Z1-Z3) vs Incoherent (Z4) topics from high-resource (S1-S3) and low-resource (T ) texts modeling (TM) still remains challenging especially in the sparse-data setting, especially for the cases where word co-occurrence data is insufficient, e.g., on short-text or a corpus of few documents. It leads to a poor quality of topics and representations. To address data sparsity issues, several works (Das et al., 2015; Nguyen et al., 2015; Gupta et al., 2019a, 2020) have introduced external knowl- edge in traditional topic models, e.g., incorporat- ing word embeddings obtained from Glove (Pen- nington et al., 2014) or word2vec (Mikolov et al., 2013a). However, no prior work in topic modeling has employed multi-view embedding spaces: (1) pretrained topics, i.e., topical embeddings obtained from large document …

Content truncated to 500 words