Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2748 - 2760 December 7-11, 2022 ©2022 Association for Computational Linguistics Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning Xiaobao Wu‡ Anh Tuan Luu‡ Xinshuai Dong† ‡Nanyang Technological University †Carnegie Mellon University xiaobao002@e.ntu.edu.sg, anhtuan.luu@ntu.edu.sg xinshuad@andrew.cmu.edu Abstract To overcome the data sparsity issue in short text topic modeling, existing methods com- monly rely on data augmentation or the data characteristic of short texts to introduce more word co-occurrence information. However, most of them do not make full use of the aug- mented data or the data characteristic: they insufficiently learn the relations among sam- ples in data, leading to dissimilar topic dis- tributions of semantically similar text pairs. To better address data sparsity, in this paper we propose a novel short text topic modeling framework, Topic-Semantic Contrastive Topic Model (TSCTM). To sufficiently model the relations among samples, we employ a new contrastive learning method with efficient pos- itive and negative sampling strategies based on topic semantics. This contrastive learning method refines the representations, enriches the learning signals, and thus mitigates the sparsity issue. Extensive experimental results show that our TSCTM outperforms state-of- the-art baselines regardless of the data aug- mentation availability, producing high-quality topics and topic distributions. 1 1 Introduction Topic models aim to discover the latent topics of a document collection and infer the topic distribu- tion of each document in an unsupervised fashion (Blei et al., 2003). Due to the effectiveness and interpretability, topic models have been popular for decades with various downstream applications (Ma et al., 2012; Mehrotra et al., 2013; Boyd-Graber et al., 2017). However, despite the success on long texts, current topic models generally cannot handle well short texts, such as tweets, headlines, and com- ments (Yan et al., 2013). The reason lies in that topic models rely on word co-occurrence informa- tion to infer latent topics, but such information is 1Our code is available at https://github.com/ bobxwu/TSCTM. x(i) april fool’s jokes range from hilarious to disastrous x(i) + april sucker ’ s laugh range from hilarious to disastrous x(ℓ) april fools’ joke goes wrong for cleveland woman x(j) should airlines create separate sections for kids, larger fliers? (a) Examples of short texts. x(i) + x(ℓ) x(j) x(i) 0.479 0.377 0.922 (b) NQTM x(i) + x(ℓ) x(j) x(i) 0.995 0.984 0.078 (c) TSCTM Figure 1: (a): Examples of short texts from TagMyNews title dataset. Text x(i) + is an augmented view of x(i), and x(i) and x(ℓ) are relevant while x(i) and x(j) are irrelevant. (b, c): Heat map of cosine simi- larity between learned topic distributions. The similari- ties of our TSCTM are more reasonable than NQTM. extremely scarce in short texts (Qiang et al., 2020). This issue, referred to as data sparsity, can hin- der state-of-the-art topic models from discovering high-quality topics and thus has attracted much attention. To overcome the data sparsity issue, traditional wisdom can be mainly categorized into two lines: (i) Augment datasets with …

Content truncated to 500 words