Neural Topic Modeling with Cycle-Consistent Adversarial Training

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9018–9030, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9018 Neural Topic Modeling with Cycle-Consistent Adversarial Training Xuemeng Hu∗ Rui Wang∗ Deyu Zhou† Yuxuan Xiong School of Computer Science and Engineering, Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, China {xuemenghu,rui wang,d.zhou,yuxuanxiong}@seu.edu.cn Abstract Advances on deep generative models have attracted significant research interest in neu- ral topic modeling. The recently proposed Adversarial-neural Topic Model models top- ics with an adversarially trained generator net- work and employs Dirichlet prior to capture the semantic patterns in latent topics. It is ef- fective in discovering coherent topics but un- able to infer topic distributions for given docu- ments or utilize available document labels. To overcome such limitations, we propose Topic Modeling with Cycle-consistent Adversarial Training (ToMCAT) and its supervised version sToMCAT. ToMCAT employs a generator net- work to interpret topics and an encoder net- work to infer document topics. Adversarial training and cycle-consistent constraints are used to encourage the generator and the en- coder to produce realistic samples that coordi- nate with each other. sToMCAT extends ToM- CAT by incorporating document labels into the topic modeling process to help discover more coherent topics. The effectiveness of the proposed models is evaluated on unsuper- vised/supervised topic modeling and text clas- sification. The experimental results show that our models can produce both coherent and in- formative topics, outperforming a number of competitive baselines. 1 Introduction Topic models, such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003), aim to discover under- lying topics and semantic structures from text col- lections. Due to its interpretability and effective- ness, LDA has been extended to many Natural Language Processing (NLP) tasks (Lin and He, 2009; McAuley and Leskovec, 2013; Zhou et al., 2017). Most of these models employ mean-field ∗Equal contribution. †Corresponding author. variational inference or collapsed Gibbs sampling (Griffiths and Steyvers, 2004) for model inference as a result of their intractable posteriors. However, such inference algorithms are model specific and require dedicated derivations. To address such limitation, neural topic models with black-box inference have been explored, with more flexible training schemes. Inspired by vari- ational autoencoder (VAE) (Kingma and Welling, 2013), Miao et al. (2016) proposed Neural Varia- tional Document Model which interprets the latent code in VAE as topics. Following this way, Srivas- tava and Sutton (2017) adopted the logistic normal prior rather than Gaussian to mimic the simplex properties of topic distribution. Logistic normal is a Laplace approximation to the Dirichlet distri- bution (MacKay, 1998). However, logistic normal can not exhibit multiple peaks at the vertices of the simplex as the Dirichlet distribution. Therefore, it is less capable of capturing the multi-modality which is crucial for topic modeling (Wallach et al., 2009). To overcome such limitation, Wang et al. (2019a) proposed Adversarial-neural Topic Model (ATM), a topic model based on Generative Adversarial Net- works (GANs) (Goodfellow et al., 2014) and sam- pling topics directly from the Dirichlet …

Content truncated to 500 words