Sparse Parallel Training of Hierarchical Dirichlet Process Topic Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2925–2934, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2925 Sparse Parallel Training of Hierarchical Dirichlet Process Topic Models Alexander Terenin Imperial College London M˚ans Magnusson Uppsala University and Aalto University Leif Jonsson Ericsson AB and Link¨oping University Abstract To scale non-parametric extensions of proba- bilistic topic models such as Latent Dirichlet allocation to larger data sets, practitioners rely increasingly on parallel and distributed systems. In this work, we study data-parallel training for the hierarchical Dirichlet process (HDP) topic model. Based upon a representation of certain conditional distributions within an HDP, we propose a doubly sparse data-parallel sampler for the HDP topic model. This sampler utilizes all available sources of sparsity found in natural language—an important way to make compu- tation efficient. We benchmark our method on a well-known corpus (PubMed) with 8m documents and 768m tokens, using a single multi-core machine in under four days. 1 Introduction Topic models are a widely-used class of methods that allow practitioners to identify latent semantic themes in large bodies of text in an unsupervised manner. They are particularly attractive in areas such as history (Yang et al., 2011; Wang et al., 2012), sociology (DiMaggio et al., 2013), and po- litical science (Roberts et al., 2014), where a desire for careful control of structure and prior informa- tion incorporated into the model motivates one to adopt a Bayesian approach to learning. In these areas, large corpora such as newspaper archives are becoming increasingly available (Ehrmann et al., 2020), and models such as latent Dirichlet alloca- tion (LDA) (Blei et al., 2003) and its nonparametric extensions (Teh et al., 2006; Teh, 2006; Hu and Boyd-Graber, 2012; Paisley et al., 2015) are widely used by practitioners. Moreover, these models are emerging as a component of data-efficient language models (Guo et al., 2020). Training topic models efficiently entails two requirements. 1. Expose sufficient parallelism that can be taken advantage of by the hardware. 2. Utilize sparsity found in natural language to control memory requirements and computa- tional complexity. In this work, we focus on the hierarchical Dirich- let process (HDP) topic model of Teh et al. (2006), which we review in Section 2. This model is a sim- ple non-trivial extension of LDA to the nonparamet- ric setting. This parallel implementation provides a blueprint for designing massively parallel train- ing algorithms in more complicated settings, such as nonparametric dynamic topic models (Ahmed and Xing, 2010) and tree-based extensions (Hu and Boyd-Graber, 2012). Parallel approaches to training HDPs have been previously introduced by a number of authors, in- cluding Newman et al. (2009), Wang et al. (2011), Williamson et al. (2013), Chang and Fisher (2014) and Ge et al. (2015). These techniques suit various settings: some are designed to explicitly incorpo- rate sparsity present in natural language and other discrete spaces, while others are intended for HDP- based continuous mixture models. Gal and Ghahra- mani (2014) have pointed out that some methods can suffer …

Content truncated to 500 words