TAN-NTM: Topic Attention Networks for Neural Topic Modeling

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3865–3880 August 1–6, 2021. ©2021 Association for Computational Linguistics 3865 TAN-NTM: Topic Attention Networks for Neural Topic Modeling Madhur Panwar2∗†, Shashank Shailabh3∗†, Milan Aggarwal1∗, Balaji Krishnamurthy1 Media and Data Science Research Labs, Adobe1 Birla Institute of Technology and Science, Pilani (BITS Pilani), India2 Indian Institute of Technology Kanpur (IIT Kanpur), India3 mdrpanwar@gmail.com, shailabhshashank@gmail.com Abstract Topic models have been widely used to learn text representations and gain insight into doc- ument corpora. To perform topic discovery, most existing neural models either take docu- ment bag-of-words (BoW) or sequence of to- kens as input followed by variational inference and BoW reconstruction to learn topic-word distribution. However, leveraging topic-word distribution for learning better features dur- ing document encoding has not been explored much. To this end, we develop a framework TAN-NTM, which processes document as a se- quence of tokens through a LSTM whose con- textual outputs are attended in a topic-aware manner. We propose a novel attention mech- anism which factors in topic-word distribution to enable the model to attend on relevant words that convey topic related cues. The output of topic attention module is then used to carry out variational inference. We perform exten- sive ablations and experiments resulting in ∼ 9 - 15 percentage improvement over score of existing SOTA topic models in NPMI coher- ence on several benchmark datasets - 20News- groups, Yelp Review Polarity and AGNews. Further, we show that our method learns bet- ter latent document-topic features compared to existing topic models through improvement on two downstream tasks: document classifi- cation and topic guided keyphrase generation. 1 Introduction Topic models (Steyvers and Griffiths, 2007) have been popularly used to extract abstract topics which occur commonly across documents in a corpus. Each topic is interpreted as a group of semantically coherent words that represent a common concept. In addition to gaining insights from unstructured texts, topic models have been used in several tasks ∗equal contribution †work done during summer internship at Adobe of practical importance such as learning text repre- sentations for document classification (Nan et al., 2019), keyphrase extraction (Wang et al., 2019b), understanding reviews for e-commerce recommen- dations (Jin et al., 2018), semantic similarity detec- tion between texts (Peinelt et al., 2020) etc. Early works on topic discovery include statis- tical methods such as Latent Semantic Analysis (Deerwester et al., 1990), Latent Dirichlet Alloca- tion (LDA) (Blei et al., 2003) which approximates each topic as a probability distribution over word vocabulary (known as topic-word distribution) and performs approximate inference over document- topic and topic-word distributions through Varia- tional Bayes. This was followed by Markov Chain Monte Carlo (MCMC) (Andrieu et al., 2003) based inference algorithm - Collapsed Gibbs sampling (Griffiths and Steyvers, 2004). These methods re- quire an expensive iterative inference step which has to be performed for each document. This was circumvented through introduction of deep neu- ral networks and Variational Autoencoders …

Content truncated to 500 words