Advancing Topical Text Classification: A Novel Distance-Based Method with Contextual Embeddings

Proceedings of Recent Advances in Natural Language Processing, pages 586–597 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_064 586 Advancing Topical Text Classification: A Novel Distance-Based Method with Contextual Embeddings Andriy Kosar Textgain & University of Antwerp (CLiPS) Antwerp, Belgium andrew@textgain.com Guy De Pauw Textgain Antwerp, Belgium guy@textgain.com Walter Daelemans University of Antwerp (CLiPS) Antwerp, Belgium walter.daelemans @uantwerpen.be Abstract This study introduces a new method for distance-based unsupervised topical text clas- sification using contextual embeddings. The method applies and tailors sentence embed- dings for distance-based topical text classifica- tion. This is achieved by leveraging the seman- tic similarity between topic labels and text con- tent, and reinforcing the relationship between them in a shared semantic space. The proposed method outperforms a wide range of existing sentence embeddings on average by 35%. Pre- senting an alternative to the commonly used transformer-based zero-shot general-purpose classifiers for multiclass text classification, the method demonstrates significant advantages in terms of computational efficiency and flexibil- ity, while maintaining comparable or improved classification results. 1 Introduction Topical text classification remains an important task in text classification because it allows users to ex- plore, analyze and organize large text collections. However, the nature of topical text classification is subjective as the content and context of the text are often perceived differently based on the intended audience. To address this, methods that dynami- cally explore topics are necessary, one of which is unsupervised text classification. This approach allows classifying text collections based on a pre- defined list of topics for further analysis. Yin et al. (2019) outlined three primary tech- niques for unsupervised text classification: 1) eval- uating the frequency of class labels in a text, 2) measuring the distance between class labels and text in a shared vector space, and 3) leveraging nat- ural language inference with pre-trained classifiers to ascertain if a class label can be deduced from the text. With the advancement of transformer models, the latter method has gained increasing attention in the NLP community due to its successful outcomes (Yin et al., 2019; Ding et al., 2022). In this study we show that task-specific sentence embeddings trained on transformer models for distance-based topical text classification, can provide a flexible and efficient alternative to the aforementioned methods. In this study, we undertake a comprehensive ex- amination of unsupervised topical text classifica- tion utilizing contextual embeddings, and propose a methodology for generating sentence embeddings that are more appropriate for this task. To achieve this objective, we first evaluate a diverse array of existing contextual embeddings and their derived sentence embeddings on seven datasets across a broad spectrum of genres and topics. Subsequently, we explore the various options for training custom sentence embeddings, including the choice of train- ing data, base models, and loss functions, with the aim of identifying the most suitable configuration for the given task. Finally, we assess the benefits and limitations of our proposed method. The paper unfolds as follows: Section 2 outlines the previous research on unsupervised text classi- fication; Section 3 presents the proposed …

Content truncated to 500 words