Towards Reinterpreting Neural Topic Models via Composite Activations

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3688 - 3703 December 7-11, 2022 ©2022 Association for Computational Linguistics Towards Reinterpreting Neural Topic Models via Composite Activations Jia Peng Lim Singapore Management University jiapeng.lim.2021@smu.edu.sg Hady W. Lauw Singapore Management University hadywlauw@smu.edu.sg Abstract Most Neural Topic Models (NTM) use a vari- ational auto-encoder framework producing K topics limited to the size of the encoder’s output. These topics are interpreted through the selec- tion of the top activated words via the weights or reconstructed vector of the decoder that are directly connected to each neuron. In this pa- per, we present a model-free two-stage process to reinterpret NTM and derive further insights on the state of the trained model. Firstly, build- ing on the original information from a trained NTM, we generate a pool of potential candi- date “composite topics” by exploiting possible co-occurrences within the original set of top- ics, which decouples the strict interpretation of topics from the original NTM. This is followed by a combinatorial formulation to select a final set of composite topics, which we evaluate for coherence and diversity on a large external cor- pus. Lastly, we employ a user study to derive further insights on the reinterpretation process. 1 Introduction To help us understand the latent structures within a text corpus, topic models associate each document with “topics” (Blei et al., 2003). In turn, each topic is associated with a set of words that frequently co-occur together in various documents, forming a semantically coherent grouping that fosters inter- pretability. Aside from the common applications in text analysis and classifications, topic models are also used in advanced downstream tasks such as in summarization (Wang et al., 2020), text generation (Wang et al., 2019), and language modelling (Lau et al., 2017). While earlier topic models are based on graphical models, more recent topic models are neural, with several based on the variational auto- encoder framework (Kingma and Welling, 2014). Traditionally, what constitutes a topic is a neuron at the encoder’s output. Its association with words is typically derived from a selection of the top acti- vated words via the weights or reconstructed vector of the decoder connected to that neuron, forming what we now interpret as a topic-word distribution. Motivation. While such autoencoder-based topic models are adept at learning lower- dimensional representations of documents, we question the notion of one-to-one correspondence between a topic and a neuron. We postulate that this traditional view belies the natural working order of a neural model, whereby it is the joint activation of several neurons, rather than the singular activation of an independent neuron, that may be responsible for the generation or reconstruction of document se- mantics. Moreover, the traditional interpretation of only the top activations in the resultant topic-word distribution ignores the potential information that might be gleaned from the rest of the distribution. We therefore hypothesize that individual neurons are but components of a “topic” that is inherently compositional in nature. And, to properly interpret …

Content truncated to 500 words