DocSCAN: Unsupervised Text Classification via Learning from Neighbors

DocSCAN: Unsupervised Text Classification via Learning from Neighbors Dominik Stammbach ETH Zurich dominsta@ethz.ch Elliott Ash ETH Zurich ashe@ethz.ch Abstract We introduce DocSCAN, a completely unsu- pervised text classification approach built on the Semantic Clustering by Adopting Nearest- Neighbors algorithm. For each document, we obtain semantically informative vectors from a large pre-trained language model. We find that similar documents have proximate vectors, so neighbors in the representation space tend to share topic labels. Our learnable cluster- ing approach then uses pairs of neighboring datapoints as a weak learning signal to auto- matically learn topic assignments. On three different text classification benchmarks, we im- prove on various unsupervised baselines by a large margin. 1 Introduction ”What is this about?” is the starting question in hu- man and machine reading of text documents. While this question would invite a variety of answers for documents in general, there is a large set of corpora for which each document can be labeled as belong- ing to a singular category or topic. Text classifica- tion is the task of automatically mapping texts into these categories. In the standard supervised setting (Vapnik, 2000), machine learning algorithms learn such a mapping from annotated examples. Anno- tating data is costly, however, and the resulting annotations are usually domain-specific. Unsuper- vised methods promise to reduce the number of labeled examples needed or to dispense with them altogether. This paper builds on recent developments in the domain of unsupervised neighbor-based clustering of images, the SCAN algorithm: Semantic Cluster- ing by Adopting Nearest neighbors (Van Gansbeke et al., 2020). We adapt the algorithm to text classi- fication and report strong experimental results on three text classification benchmarks. The intuition behind SCAN is that images often share the same label, if their embeddings in some representation space are close to each other. Thus, we can lever- age this regularity as a weakly supervised signal for training models. We encode a datapoint and its neighbors through a network where the output of the network is determined by a classification layer. The model learns that it should assign simi- lar output probabilities to a datapoint and each of its neighbors. In the ideal case, model output is consis- tent and one-hot, i.e. the model confidently assigns the same label to two neighboring datapoints. Deep Transformer networks have led to rapid improvements in text classification and other natu- ral language processing (NLP) tasks (see e.g. Yang et al., 2019). We draw from such models to obtain task-agnostic contextualized language representa- tions. We use SBERT embeddings (Reimers and Gurevych, 2019), which have proven performance in a variety of downstream tasks, such as retrieving semantically similar documents and text clustering. We show that in this semantic space, indeed neigh- boring documents tend to often share the same class label and we can use this proximity to build a dataset on which we apply our neighbor-based clustering objective. We find that training a model exploiting this regularity works well for text classi- fication and outperforms a standard unsupervised baseline …

Content truncated to 500 words