Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3886 - 3893 July 10-15, 2022 ©2022 Association for Computational Linguistics Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics Zihan Zhang1, Meng Fang2, Ling Chen1, Mohammad-Reza Namazi-Rad3 1AAII, University of Technology Sydney, NSW, Australia Zihan.Zhang-5@student.uts.edu.au, Ling.Chen@uts.edu.au 2Eindhoven University of Technology, Eindhoven, the Netherlands m.fang@tue.nl 3NIASRA, University of Wollongong, NSW, Australia mrad@uow.edu.au Abstract Recent work incorporates pre-trained word em- beddings such as BERT embeddings into Neu- ral Topic Models (NTMs), generating highly coherent topics. However, with high-quality contextualized document representations, do we really need sophisticated neural models to obtain coherent and interpretable topics? In this paper, we conduct thorough experiments showing that directly clustering high-quality sentence embeddings with an appropriate word selecting method can generate more coherent and diverse topics than NTMs, achieving also higher efficiency and simplicity.1 1 Introduction Topic modelling is an unsupervised method to un- cover latent semantic themes among documents (Boyd-Graber et al., 2017). Neural topic mod- els (NTMs) (Miao et al., 2016; Srivastava and Sutton, 2017) incorporating neural components have significantly advanced the modelling results than the traditional Latent Dirichlet Allocation (LDA; Blei et al. 2001). Later, contextualized word and sentence embeddings produced by pre- trained language models such as BERT (Devlin et al., 2019) have demonstrated the state-of-the-art results in multiple Natural Language Processing (NLP) tasks (Xia et al., 2020), which attracts atten- tions from the topic modelling community. Recent work has successfully incorporated these contextu- alized embeddings into NTMs, showing improved topic coherence than conventional NTMs that use Bag-of-Words (BoW) as document representations (Bianchi et al., 2021a,b; Jin et al., 2021). Despite the promising performance, existing NTMs are gen- erally based on a variational autoencoder frame- work (VAE; Kingma and Welling 2014), which suffers from hyper-parameters tuning and compu- tational overheads (Zhao et al., 2021). Moreover, 1Code is available at https://github.com/ hyintell/topicx the integration of the pre-trained embeddings to the standard VAE framework adds additional model complexity. With high-quality contextualized doc- ument representations, do we really need sophisti- cated NTMs to obtain coherent and interpretable topics? Recent work (Aharoni and Goldberg, 2020; Sia et al., 2020; Thompson and Mimno, 2020; Grooten- dorst, 2020) has shown that directly congregating contextualized embeddings can get semantically similar word or document clusters. Specifically, Sia et al. (2020) cluster vocabulary-level word em- beddings and obtain top words from each cluster using weighing and re-ranking, while Thompson and Mimno (2020) consider polysemy and perform token-level clustering. However, the use of term frequency (TF) to select topic words fails to cap- ture the semantics of clusters precisely because words with high frequency may be common across different clusters. Grootendorst (2020) propose a class-based Term Frequency- Inverse Document Frequenc (c-TF-IDF) method that extract impor- tant words from each clustered documents, which tends to choose representative words within each cluster to form topics. However, it overlooks the global semantics between clusters which could be …

Content truncated to 500 words