Learning VAE-LDA Models with Rounded Reparameterization Trick

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1315–1325, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1315 Learning VAE-LDA Models with Rounded Reparameterization Trick Runzhi Tian School of EECS University of Ottawa, Canada rtian081@uottawa.ca Yongyi Mao School of EECS University of Ottawa, Canada ymao@uottawa.ca Richong Zhang BDBC and SKLSDE Beihang University, China zhangrc@act.buaa.edu.cn Abstract The introduction of VAE provides an efficient framework for the learning of generative mod- els, including generative topic models. How- ever, when the topic model is a Latent Dirich- let Allocation (LDA) model, a central tech- nique of VAE, the reparameterization trick, fails to be applicable. This is because no repa- rameterization form of Dirichlet distributions is known to date that allows the use of the repa- rameterization trick. In this work, we propose a new method, which we call Rounded Repa- rameterization Trick (RRT), to reparameterize Dirichlet distributions for the learning of VAE- LDA models. This method, when applied to a VAE-LDA model, is shown experimentally to outperform the existing neural topic models on several benchmark datasets and on a synthetic dataset. 1 Introduction Probabilistic generative models are widely used in topic modelling and have achieved great success in many applications (Deerwester et al., 1990)(Hof- mann, 1999)(Blei et al., 2003)(Blei and Lafferty, 2006). A landmark of topic models is Latent Dirich- let Allocation (LDA) (Blei et al., 2003), where a document is treated as a bag of words and each word is modelled via a generative process. More specifically, in this generative process, a topic dis- tribution is first drawn from a Dirichlet prior, then a topic is sampled from the topic distribution and a word is drawn subsequently from the word distri- bution corresponding to the drawn topic. Since its introduction, LDA has shown great power in a large varieties of natural language applications (Wei and Croft, 2006)(AlSumait et al., 2008)(Mehrotra et al., 2013). However, the classical methods of learning LDA, such as variational techniques and collapsed Gibbs sampling, entails high computation complex- ity in posterior inference(Blei et al., 2003)(Grif- fiths and Steyvers, 2004), which limits the ability of LDA on modelling large corpus. Variational AutoEncoder (VAE) or AutoEn- coding Variational Bayes (AEVB) (Kingma and Welling, 2013) provides another choice of learning a generative model. Under the VAE framework, a generative model is specified by first drawing a latent vector z from a prior distribution and then transforming this vector through a neural network, called decoder, which subsequently generates the observation x. Using a variational inference ap- proach, VAE couples the decoder network with another network, called encoder, responsible for computing the posterior distribution of the latent variable z for each observation x. A key technique of VAE is its “reparameterization trick”, in which sampling from the posterior is performed by sam- pling a noise variable ϵ from some distribution p(ϵ) and then transforming ϵ to z using a differentiable function. This technique allows the model to be trained efficiently using back propagation. The VAE framework significantly alleviates …

Content truncated to 500 words