Unified Neural Topic Model via Contrastive Learning and Term Weighting

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1802–1817 May 2-6, 2023 ©2023 Association for Computational Linguistics Unified Neural Topic Model via Contrastive Learning and Term Weighting Sungwon Han1 Mingi Shin1 Sungkyu Park2∗ Changwook Jung1 Meeyoung Cha3,1∗ 1School of Computing, KAIST, South Korea 2Department of AI Convergence, Kangwon National University, South Korea 3Data Science Group, Institute for Basic Science (IBS), South Korea {lion4151, mingi.shin, wookidoki}@kaist.ac.kr, shaun@kangwon.ac.kr, mcha@ibs.re.kr Abstract Two types of topic modeling predominate: gen- erative methods that employ probabilistic la- tent models and clustering methods that iden- tify semantically coherent groups. This paper newly presents UTopic (Unified neural Topic model via contrastive learning and term weight- ing) that combines the advantages of these two types. UTopic uses contrastive learning and term weighting to learn knowledge from a pre- trained language model and discover influential terms from semantically coherent clusters. Ex- periments show that the generated topics have a high-quality topic-word distribution in terms of topic coherence, outperforming existing base- lines across multiple topic coherence measures. We demonstrate how our model can be used as an add-on to existing topic models and improve their performance. 1 Introduction One of the most common tasks in natural language processing (NLP) is to find cohesive topics in a text corpus. Topic modeling is widely used in var- ious applications, including trend extraction from real-time streams such as social media (Lau et al., 2012; Park et al., 2021) and identification of no- table events upon risk (Shin et al., 2020). Two representative lines of work exist: generative meth- ods and clustering-based methods. Generative methods follow the assumption of probabilistic latent semantic analysis (hereafter p- LSA) such that every word token in a document is sampled from a mixture of latent topics (Hoffman, 1999). Such methods estimate both the latent topic distribution per document and word distribution per topic from Bag-of-Words (BoW) representa- tion (Blei et al., 2003; Fisher et al., 2020; Miao et al., 2017; Yan et al., 2013). Latent Dirichlet Allo- cation (LDA), for example, uses a Bayesian model to estimate the latent topic distribution (Blei et al., ∗Corresponding Authors 2003). Variational Autoencoder (VAE) retrieves latent topics from the topic distribution while also recovering the original input (Srivastava and Sut- ton, 2017). Recent techniques have employed pretrained language models on top of VAE-based models to improve topic quality (Bianchi et al., 2021a,b). Generative probabilistic models work on the assumption that each topic is a mix of words from a larger set. However, they share a common weakness: the BoW representation only contains word-level co-occurrences and fails to capture each word token’s importance in the document’s context information. Consequently, the quality of the esti- mated topic is reliant on the choice of the word set. Clustering methods, on the other hand, regard topics as semantic clusters discovered over the doc- ument or word embedding space (Angelov, 2020; Grootendorst, 2022; Sia et al., 2020). They utilize the knowledge of the pretrained language model (e.g., BERT (Devlin et …

Content truncated to 500 words