An Empirical Analysis of Topic Models: Uncovering the Relationships between Hyperparameters, Document Length and Performance Measures

Proceedings of Recent Advances in Natural Language Processing, pages 1408–1416 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_157 1408 An Empirical Analysis of Topic Models: Uncovering the Relationships between Hyperparameters, Document Length and Performance Measures Silvia Terragni, Elisabetta Fersini University of Milano-Bicocca Viale Sarca 336, 20126 Milan, Italy s.terragni4@campus.unimib.it, elisabetta.fersini@unimib.it Abstract Neural Topic Models are recent neural models that aim at extracting the main themes from a collection of documents. The comparison of these models is usually limited because the hy- perparameters are held fixed. In this paper, we present an empirical analysis and comparison of Neural Topic Models by finding the optimal hyperparameters of each model for four differ- ent performance measures adopting a single- objective Bayesian optimization. This allows us to determine the robustness of a topic model for several evaluation metrics. We also empir- ically show the effect of the length of the doc- uments on different optimized metrics and dis- cover which evaluation metrics are in conflict or agreement with each other. 1 Introduction Topic models (Blei, 2012) are probabilistic genera- tive models that aim at identifying the underlying themes, or topics, in a collection of documents. Al- though they are used in a vast range of applications, from text exploratory purposes to information re- trieval tasks (Boyd-Graber et al., 2017), most of the investigations disregard the elements that influ- ence the results generated by the models and, in particular, what is the effect on their performance. Several works explore topic modeling over a range of different models, topics, and measures, but usually focus on classical topic models (Greene et al., 2014; Stevens et al., 2012), e.g. Latent Dirichlet Allocation (LDA) (Blei et al., 2003) or Non-negative Matrix Factorization (NMF) (Lee and Seung, 2000), and solely on a single evalu- ation measure (Stevens et al., 2012; O’Callaghan et al., 2015). Doan and Hoang recently made an effort to benchmark neural topic models, however, the au- thors seem to disregard the importance of the hyper- parameter selection. In fact, the evaluations of topic models are usually limited to the comparison of models whose hyperparameters are fixed. Yet, the hyperparameters that control the models can have a great impact on their performance. Therefore, fix- ing them prevents the researchers from discovering the best topic model on a given dataset. In the latest years, Neural Topic Models (NTM) (Zhao et al., 2021; Dieng et al., 2020; Bianchi et al., 2021a,b) have gained popularity, due to their flexibility and scalability. The problem of finding the best hy- perparameter configuration has become even more compelling, since topic models based on neural networks are usually controlled by a high number of hyperparameters. In this paper, we perform an empirical analysis of recent NTMs by optimizing the hyperparameters of the models with respect to different metrics. We aim to investigate if there exists a potential relation- ship between hyperparameters, document length and performance measures, to finally understand under which conditions we can exploit at best the potentiality of each model. In particular, the fol- lowing research questions …

Content truncated to 500 words