Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling Decoder

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1772–1782, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1772 Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling Decoder Xiaobao Wu1 Chunping Li1 Yan Zhu2 Yishu Miao3 1Tsinghua University 2Southwest Jiaotong University 3Imperial College London wxb18@mails.tsinghua.edu.cn, cli@mail.tsinghua.edu.cn yzhu@swjtu.edu.cn, y.miao20@imperial.ac.uk Abstract Topic models have been prevailing for many years on discovering latent semantics while modeling long documents. However, for short texts they generally suffer from data spar- sity because of extremely limited word co- occurrences; thus tend to yield repetitive or trivial topics with low quality. In this paper, to address this issue, we propose a novel neu- ral topic model in the framework of autoen- coding with a new topic distribution quantiza- tion approach generating peakier distributions that are more appropriate for modeling short texts. Besides the encoding, to tackle this issue in terms of decoding, we further pro- pose a novel negative sampling decoder learn- ing from negative samples to avoid yielding repetitive topics. We observe that our model can highly improve short text topic modeling performance. Through extensive experiments on real-world datasets, we demonstrate our model can outperform both strong traditional and neural baselines under extreme data spar- sity scenes, producing high-quality topics. 1 Introduction In addition to formal documents, short texts play an increasingly more important role in the era of information explosion where people could instantly share ideas, feelings, and comments via short text fragments, including tweets, headlines, and product reviews, etc. The latent semantics or topics discov- ered among these short texts can be utilized in many applications, such as content summarization (Ma et al., 2012), classification(Zeng et al., 2018a), and recommendations (Zeng et al., 2018b; Mehrotra et al., 2013). However, conventional topic models (Blei et al., 2003) work reasonably well on vari- ous kinds of long documents, but perform poorly on short texts. The main underlying reason is that the co-occurrence information from short texts is extremely limited as known as the data sparsity sports scores games soccer league tennis ncaa players football sports tennis soccer hockey games football beach match players sports match cup hockey olympic football players sport league sports football sport league games tennis champions club sports football league game tennis players hockey games scores bad additional abstract aspectj behave displayed customise accept abstract behave accept additional bad displayed customise abstract accept behave additional adding long many administration Table 1: Repetitive and trivial topics from short texts. Repetitive words are underlined. problem which hinders the topic models from learn- ing effective semantics and high-quality topics in a pure unsupervised learning fashion. Therefore, several approaches have been proposed to allevi- ate this issue. One simple approach is to yield pseudo texts (Quan et al., 2015), so that the con- ventional topic models can apply, e.g., user data (Weng et al., 2010), hashtags (Mehrotra et al., 2013) and external corpora (Zuo et al., 2016), but aux- iliary information is not always available. In an- other vein, extra …

Content truncated to 500 words