Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modeling

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2220–2229 May 2-6, 2023 ©2023 Association for Computational Linguistics Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modeling Suman Adhya, Avishek Lahiri, Debarshi Kumar Sanyal Indian Association for the Cultivation of Science, Jadavpur, Kolkata-700032, India { adhyasuman30, avisheklahiri2014, debarshisanyal }@gmail.com Abstract Dropout is a widely used regularization trick to resolve the overfitting issue in large feedfor- ward neural networks trained on a small dataset, which performs poorly on the held-out test sub- set. Although the effectiveness of this regular- ization trick has been extensively studied for convolutional neural networks, there is a lack of analysis of it for unsupervised models and in particular, VAE-based neural topic models. In this paper, we have analyzed the consequences of dropout in the encoder as well as in the de- coder of the VAE architecture in three widely used neural topic models, namely, contextual- ized topic model (CTM), ProdLDA, and em- bedded topic model (ETM) using four publicly available datasets. We characterize the dropout effect on these models in terms of the quality and predictive performance of the generated topics. 1 Introduction Dropout (Hinton et al., 2012) is used while train- ing neural networks, by stochastically dropping out the activation of neurons to prevent complex co-adaptations of feature vectors (Baldi and Sad- owski, 2013). The working of dropout is attributed to the implicit averaging over an ensemble of neu- ral networks (Labach et al., 2019; Warde-Farley et al., 2014). It has been shown to be effective on supervised learning tasks to prevent overfitting (Srivastava et al., 2014). As the volume of digital documents significantly increases with time, organizing them manually is becoming quite an inconvenient task. Because of the ability of topic models to learn a thematic structure from a set of documents in an unsuper- vised manner and label the documents with their corresponding dominant topics, the significance of topic models is enormous in this area (Hall et al., 2008; Adhya and Sanyal, 2022). But in the traditional topic models, not only the computa- tion cost of the approximate posterior is very high but also for a small change in the modeling as- sumption, re-derivation of the inference method is needed. With greater flexibility and scalability than traditional topic models, a class of Neural Topic Models (NTMs) aim to leverage the potential of neural networks using the AEVB (Kingma and Welling, 2014) based inference technique. Follow- ing (Zhao et al., 2021), we refer to this class of models as VAE-NTMs where the training objective is to maximize the log-likelihood of the reconstruc- tion of the input document while minimizing the KL-divergence of the learned posterior distribution of the latent space from a known prior distribution. An earlier study by (Ha et al., 2019) of the dropout effect on two traditional topic models LDA (Blei et al., 2003) and BTM (Yan et al., 2013) shows that the correct choice of the …

Content truncated to 500 words