Neural Mixed Counting Models for Dispersed Topic Discovery

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6159–6169 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6159 Neural Mixed Counting Models for Dispersed Topic Discovery Jiemin Wu1,∗, Yanghui Rao1,†, Zusheng Zhang1, Haoran Xie2, Qing Li3, Fu Lee Wang4, Ziye Chen1 1School of Data and Computer Science, Sun Yat-sen University, Guangzhou, China 2Department of Computing and Decision Sciences, Lingnan University, Hong Kong 3Department of Computing, The Hong Kong Polytechnic University, Hong Kong 4School of Science and Technology, The Open University of Hong Kong, Hong Kong wujm29@mail2.sysu.edu.cn, raoyangh@mail.sysu.edu.cn, hrxie2@gmail.com, csqli@comp.polyu.edu.hk, pwang@ouhk.edu.hk, {zhangzsh3, chenzy35}@mail2.sysu.edu.cn Abstract Mixed counting models that use the negative binomial distribution as the prior can well model over-dispersed and hierarchically de- pendent random variables; thus they have at- tracted much attention in mining dispersed document topics. However, the existing pa- rameter inference method like Monte Carlo sampling is quite time-consuming. In this pa- per, we propose two efficient neural mixed counting models, i.e., the Negative Binomial- Neural Topic Model (NB-NTM) and the Gam- ma Negative Binomial-Neural Topic Model (GNB-NTM) for dispersed topic discovery. Neural variational inference algorithms are de- veloped to infer model parameters by using the reparameterization of Gamma distribution and the Gaussian approximation of Poisson distri- bution. Experiments on real-world datasets in- dicate that our models outperform state-of-the- art baseline models in terms of perplexity and topic coherence. The results also validate that both NB-NTM and GNB-NTM can produce explainable intermediate variables by generat- ing dispersed proportions of document topics. 1 Introduction Mixture modeling is an essential topic in statistics and machine learning areas, owing to generating the random probability measure of data samples belonging to multiple clusters. In unsupervised learning tasks such as topic discovery, mixture modeling has gained increasing attention from re- searchers (Wang et al., 2011; Zhou and Carin, 2012, 2015; Zhou, 2018; Zhao et al., 2019). Specifically, mixture modeling over document words devotes to assign these words to different topics via ran- dom probability measures. Hierarchical Dirichlet ∗ The first two authors contributed equally to this work which was finished when Jiemin Wu was an undergraduate student of his final year. †The corresponding author. Process (HDP) (Teh et al., 2004) is one of the repre- sentative methods in mixture modeling, which can characterize the two-level dependency of random probability measures. Although we can use Monte Carlo sampling or variational inference to estimate the parameters in HDP, it requires the help of indi- rect construction of random variables such as the Chinese Restaurant Franchise (Teh et al., 2004) or the Stick-Breaking construction (Wang et al., 2011) due to the lack of conjugation between the two-tier Dirichlet processes. This makes the inference of HDP mostly complicated (Zhou et al., 2016). The mixed counting models represented by the Negative Binomial (NB) process (Titsias, 2007) and the Gamma Negative Binomial (GNB) process (Zhou and Carin, 2012) have solved this problem to a certain extent, in which, the normalized GNB process has been proven to be equivalent to HDP (Zhou and …

Content truncated to 500 words