Pre-training and Fine-tuning Neural Topic Model: A Simple yet Effective Approach to Incorporating External Knowledge

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5980 - 5989 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Pre-training and Fine-tuning Neural Topic Model: A Simple yet Effective Approach to Incorporating External Knowledge Linhai Zhang1 Xumeng Hu1 Boyu Wang1 Deyu Zhou1∗ Qian-Wen Zhang2 Yunbo Cao2 1 School of Computer Science and Engineering, Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, China 2 Tencent Could Xiaowei, Beijing, China {lzhang472, xuemenghu, wby1999, d.zhou}@seu.edu.cn {cowenzhang, yunbocao}@tencent.com Abstract Recent years have witnessed growing interests in incorporating external knowledge such as pre-trained word embeddings (PWEs) or pre- trained language models (PLMs) into neural topic modeling. However, we found that em- ploying PWEs and PLMs for topic modeling only achieved limited performance improve- ments but with huge computational overhead. In this paper, we propose a novel strategy to in- corporate external knowledge into neural topic modeling where the neural topic model is pre- trained on a large corpus and then fine-tuned on the target dataset. Experiments have been conducted on three datasets and results show that the proposed approach significantly outper- forms both current state-of-the-art neural topic models and some topic modeling approaches enhanced with PWEs or PLMs. Moreover, fur- ther study shows that the proposed approach greatly reduces the need for the huge size of training data. 1 Introduction Topic models have been widely used for discov- ering hidden themes from a large collection of documents in an unsupervised manner. Recently, to avoid the complex and specific inference pro- cess of graph model-based method such as LDA (Blei et al., 2003), neural topic modeling that uti- lizes neural-network-based black-box inference has been the main research direction in this field (Blei, 2012; Miao et al., 2016; Srivastava and Sutton, 2017). Typically, neural topic models infer topics of a document by utilizing its bag-of-words (BoWs) representation to capture word co-occurrence pat- terns. The BoWs representation, however, fails to encode rich word semantics, leading to relatively inferior quality of topics generated by the topic models. Therefore, approaches have been proposed to address the limitation of BoWs representation ∗Corresponding author. by incorporating the external knowledge, such as pre-trained word embeddings (PWEs) (Das et al., 2015; Wang et al., 2020; Dieng et al., 2020). In recent years, pre-trained language models (PLMs) (Peters et al., 2018; Devlin et al., 2019; Brown et al., 2020) have achieved state-of-the-art performance on a wide range of natural language processing tasks. Different from PWEs1 in which a word is mapped to a static word emebdding, PLMs generate a specific word embedding for each oc- currence of a word depending on the context. It is appealing to incorporate PLMs into topic mod- els since contextualized embeddings generated by PLMs encode richer semantics and naturally deal with word polysemy (Pasini et al., 2020). One straightforward way is to replace BoWs representa- tion with the outputs of PLM (Bianchi et al., 2020b) in existing topic models or take PLM outputs as additional inputs …

Content truncated to 500 words