A DistilBERTopic Model for Short Text Documents

A DistilBERTopic Model for Short Text Documents Junaid Rashid1, Jungeun Kim2, Usman Naseem3, Amir Hussain4 1Department of Computer Science and Engineering, Kongju National University, South Korea 2Department of Software, Kongju National University, South Korea 3School of Computer Science, University of Sydney, Australia 4School of Computing, Edinburgh Napier University, UK 1junaidrashid062@gmail.com,2jekim@kongju.ac.kr 3usman.naseem@sydney.edu.au,4a.hussain@napier.ac.uk Abstract The analysis of short text documents has be- come a vital and challenging task. Topic mod- els are utilized to extract topics from a large amount of text data. However, these topic mod- els typically suffer from data sparsity problems when applied to short texts because of relatively lower word co-occurrence patterns. As a result, they tend to provide repetitive or trivial top- ics of poor quality. Therefore, we presented a DistilBERTopic model to remove the sparsity problem and discover quality topics more accu- rately from short texts. DistilBERTopic model utilized the pre-trained transformer-based lan- guage models, reduced the dimensionality ef- fect on embedding, clustered these embeddings, and discovered the topics from short text docu- ments. Experimental results demonstrate that the DistilBERTopic model achieves better clas- sification and topic coherence than other state- of-the-art topic models for real-world datasets. 1 Introduction Numerous Web applications, including online so- cial networks, recommendation systems, and ques- tion and answer systems, have recently grown in popularity. User-generated content has prolifer- ated, particularly the massive increase in short text in various contexts like blogging, text messages, or customer reviews. It has become a crucial and difficult challenge in many applications to automat- ically discover latent semantic topics from huge amounts of short texts. Considerable effort has been devoted to tack- ling the issue of data sparsity in topic modeling for short text documents. In prior work, for instance, a method is developed for aggregation of a few spe- cific sentences that recreate a lengthier pseudo doc- ument by employing appropriate strategies like as combining all text messages originating from a sin- gle author (Hong and Davison, 2010a) or establish- ing relation information between hashtags (Wang, Liu, Qu, Huang, Chen, and Feng, 2014). In addi- tion, some brief messages can contain contextual information such as URL, location, or timestamp. A large amount of the world’s textual data comes from news sources and web portals, and all these sources often include various descriptions (Ram- age, Hall, Nallapati, and Manning, 2009). How- ever, these strategies may fail in the absence of contextual information (Naseem, Razzak, Khan, and Prasad, 2021). Conventional topic modeling techniques like Latent Semantic Analysis (LSA) (Deerwester, Dumais, Furnas, Landauer, and Harsh- man, 1990), Probabilistic Latent Semantic Analysis (PLSA) (Hofmann, 1999), and Latent Dirichlet Al- location (LDA) (Blei, Ng, and Jordan, 2001) is extensively employed for discovering the topics from documents. The formation of co-occurring word pairs across the same document is explicitly modeled in the biterm topic model (BTM) (Cheng, Yan, Lan, and Guo, 2014). The technique avoids the sparsity issue at the document level by aggregating the corpora biterms into a big pseudo document from which the topic distribution is inferred. However, …

Content truncated to 500 words