Improving Neural Topic Models using Knowledge Distillation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1752–1771, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1752 Improving Neural Topic Models using Knowledge Distillation Alexander Hoyle∗ Computer Science University of Maryland College Park, MD hoyle@umd.edu Pranav Goel∗ Computer Science University of Maryland College Park, MD pgoel1@umd.edu Philip Resnik Linguistics / UMIACS University of Maryland College Park, MD resnik@umd.edu Abstract Topic models are often used to identify human- interpretable topics to help make sense of large document collections. We use knowledge dis- tillation to combine the best attributes of proba- bilistic topic models and pretrained transform- ers. Our modular method can be straightfor- wardly applied with any neural topic model to improve topic quality, which we demon- strate using two models having disparate archi- tectures, obtaining state-of-the-art topic coher- ence. We show that our adaptable framework not only improves performance in the aggre- gate over all estimated topics, as is commonly reported, but also in head-to-head comparisons of aligned topics. 1 Introduction The core idea behind the predominant pretrain and fine-tune paradigm for transfer learning in NLP is that general language knowledge, gleaned from large quantities of data using unsupervised objec- tives, can serve as a foundation for more special- ized endeavors. Current practice involves taking the full model that has amassed such general knowl- edge and fine-tuning it with a second objective ap- propriate to the new task (see Raffel et al., 2019, for an overview). Using these methods, pre-trained transformer-based language models (e.g., BERT, Devlin et al., 2019) have been employed to great effect on a wide variety of NLP problems, thanks, in part, to a fine-grained ability to capture aspects of linguistic context (Clark et al., 2019; Liu et al., 2019; Rogers et al., 2020). However, this paradigm introduces a subtle but insidious limitation that becomes evident when the downstream application is a topic model. A topic model may be cast as a (stochastic) autoencoder (Miao et al., 2016), and we could fine-tune a pre- ∗Equal contribution. BoW BAT art chess gingerbread modernism painter picasso θd · B Base neural topic model d Marcel Duchamp was a painter, sculptor, chess player, and writer whose work is associated with Cubism, Dada, and conceptual art. Figure 1: Improving a base neural topic model with knowledge distillation. A document is mapped through both a standard BoW representation and a BERT-based Auto-encoder “Teacher” (BAT), yielding two distribu- tions over words. These are used as the ground truth in the “student” topic model’s document reconstruction loss LKD (backpropagated along the dotted line). Cru- cially, the BAT distribution assigns mass to unobserved but related terms (unbolded). trained transformer with an identical document re- construction objective. But in replacing the original topic model, we lose the property that makes it de- sirable: its interpretability. The transformer gains its contextual power from its ability to exploit a huge number of parameters, while the interpretabil- ity of a topic model comes from a dramatic dimen- sionality reduction. We combine the advantages …

Content truncated to 500 words