Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1728–1736, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1728 Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too! Suzanna Sia Ayush Dalmia Sabrina J. Mielke Department of Computer Science Johns Hopkins University Baltimore, MD, USA ssia1@jhu.edu, adalmia1@jhu.edu, sjmielke@jhu.edu Abstract Topic models are a useful analysis tool to un- cover the underlying themes within document collections. The dominant approach is to use probabilistic topic models that posit a genera- tive story, but in this paper we propose an al- ternative way to obtain topics: clustering pre- trained word embeddings while incorporating document information for weighted clustering and reranking top words. We provide bench- marks for the combination of different word embeddings and clustering algorithms, and analyse their performance under dimensional- ity reduction with PCA. The best performing combination for our approach performs as well as classical topic models, but with lower run- time and computational complexity. 1 Introduction Topic models are the standard approach for ex- ploratory document analysis (Boyd-Graber et al., 2017), which aims to uncover main themes and underlying narratives within a corpus. But in times of distributed and even contextualized embeddings, are they the only option? This work explores an alternative to topic model- ing by casting ‘key themes’ or ‘topics’ as clusters of word types under the modern distributed represen- tation learning paradigm: unsupervised pre-trained word embeddings provide a representation for each word type as a vector, allowing us to cluster them based on their distance in high-dimensional space. The goal of this work is not to strictly outperform, but rather to benchmark standard clustering of mod- ern embedding methods against the classical ap- proach of Latent Dirichlet Allocation (LDA; Blei et al., 2003). We restrict our study to influential embedding methods and focus on centroid-based clustering algorithms as they provide a natural way to obtain the top words in each cluster based on distance from the cluster center.1 Aside from reporting the best performing com- bination of word embeddings and clustering al- gorithm, we are also interested in whether there are consistent patterns: embeddings which per- form consistently well across clustering algorithms might be good representations for unsupervised document analysis, clustering algorithms that per- form consistently well are more likely to generalize to future word embedding methods. To make our approach reliably work as well as LDA, we incorporate corpus frequency statistics directly into the clustering algorithm, and quan- tify the effects of two key methods, 1) weighting terms during clustering and 2) reranking terms for obtaining the top J representative words. Our con- tributions are as follows: • We systematically apply centroid-based clus- tering algorithms on top of a variety of pre- trained word embeddings and embedding methods for document analysis. • Through weighted clustering and reranking of top words we obtain sensible topics; the best performing combination is comparable with LDA, but with smaller time complexity and empirical runtime. • We show that …

Content truncated to 500 words