Rethinking Topic Modelling: From Document-Space to Term-Space

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2250–2259 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2250 Rethinking Topic Modelling: From Document-Space to Term-Space Magnus Sahlgren Research Institutes of Sweden (RISE) Box 1263, 164 29 Kista, Sweden magnus.sahlgren@ri.se Abstract This paper problematizes the reliance on docu- ments as the basic notion for defining term in- teractions in standard topic models. As an al- ternative to this practice, we reformulate topic distributions as latent factors in term similarity space. We exemplify the idea using a number of standard word embeddings built with very wide context windows. The embedding spaces are transformed to sparse similarity spaces, and topics are extracted in standard fashion by factorizing to a lower-dimensional space. We use a number of different factorization tech- niques, and evaluate the various models using a large set of evaluation metrics, including pre- viously published coherence measures, as well as a number of novel measures that we suggest better correspond to real-world applications of topic models. Our results clearly demonstrate that term-based models outperform standard document-based models by a large margin. 1 Introduction Topic models are often used in real-world text analysis scenarios as tools for efficient data explo- ration. The typical modus operandi in such scenar- ios is to run a topic model with standard parame- ter settings on the data, and to extract some fixed number n of topics and some fixed number m of words per topic, and then manually interpret, and draw conclusions from, the resulting term lists. A common choice for both n and m is around 10. This means that the human analyst only needs to look at around 100 terms in total instead of read- ing a text collection consisting of possibly several hundreds of thousands, or even millions, of run- ning words. In terms of efficiency, this is an in- valuable tool for content analysis. Topic models extract topics by uncovering (la- tent) interactions between terms in document space. This methodology obviously assumes that data arrives with clear and consistent document boundaries, and, in the best case, a fairly even dis- tribution of number of words per document. Un- fortunately, this assumption rarely holds in real- world scenarios, where data may arrive in streams, in batches without clear document boundaries, or with very large variations in document lengths. To handle such scenarios, it would be desirable to use a model that is insensitive to the formatting of the input data. In this paper, we discuss and evaluate one such approach, which embeds the topic mod- elling process entirely in term space. This makes the model less sensitive to document-formatting, and, as it turns out, also more precise. This work is primarily motivated by the practi- cal usability of topic models in real-world analysis scenarios. In such applications – common in par- ticular in the social sciences, and in security and defence applications – the analyst only cares about the top ranked terms in the resulting term lists. We therefore introduce …

Content truncated to 500 words