Topic Modeling with Wasserstein Autoencoders

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6345–6381 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6345 Topic Modeling with Wasserstein Autoencoders Feng Nan†, Ran Ding ∗ ‡, Ramesh Nallapati†, Bing Xiang† Amazon Web Services†, Compass Inc.‡ {nanfen, rnallapa, bxiang}@amazon.com†, ran.ding@compass.com‡ Abstract We propose a novel neural topic model in the Wasserstein autoencoders (WAE) framework. Unlike existing variational autoencoder based models, we directly enforce Dirichlet prior on the latent document-topic vectors. We exploit the structure of the latent space and apply a suitable kernel in minimizing the Maximum Mean Discrepancy (MMD) to perform distri- bution matching. We discover that MMD per- forms much better than the Generative Adver- sarial Network (GAN) in matching high di- mensional Dirichlet distribution. We further discover that incorporating randomness in the encoder output during training leads to signif- icantly more coherent topics. To measure the diversity of the produced topics, we propose a simple topic uniqueness metric. Together with the widely used coherence measure NPMI, we offer a more wholistic evaluation of topic qual- ity. Experiments on several real datasets show that our model produces significantly better topics than existing topic models. 1 Introduction Probabilistic topic models (Hoffman et al., 2010) have been widely used to explore large collec- tions of documents in an unsupervised manner. They can discover the underlying themes and or- ganize the documents accordingly. The most pop- ular probabilistic topic model is the Latent Dirich- let Allocation (LDA) (Blei et al., 2003), where the authors developed a variational Bayesian (VB) al- gorithm to perform approximate inference; sub- sequently (Griffiths and Steyvers, 2004) proposed an alternative inference method using collapsed Gibbs sampling. More recently, deep neural networks have been successfully used for such probabilistic models with the emergence of variational autoencoders ∗This work was done when the author was with Amazon. (VAE) (Kingma and Welling, 2013). The key ad- vantage of such neural network based models is that inference can be carried out easily via a for- ward pass of the recognition network, without the need for expensive iterative inference scheme per example as in VB and collapsed Gibbs sampling. Topic models that fall in this framework include NVDM (Miao et al., 2016), ProdLDA (Srivastava and Sutton, 2017) and NTM-R (Ding et al., 2018). At a high level, these models consist of an en- coder network that maps the Bag-of-Words (BoW) input to a latent document-topic vector and a de- coder network that maps the document-topic vec- tor to a discrete distribution over the words in the vocabulary. They are autoencoders in the sense that the output of the decoder aims to reconstruct the word distribution of the input BoW representa- tion. Besides the reconstruction loss, VAE-based methods also minimize a KL-divergence term be- tween the prior and posterior of the latent vector distributions. Despite their popularity, these VAE- based topic models suffer from several conceptual and practical challenges. First, the Auto-Encoding Variational Bayes (Kingma and Welling, 2013) framework of VAE …

Content truncated to 500 words