Cluster & Tune: Boost Cold Start Performance in Text Classification

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7639 - 7653 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Cluster & Tune: Boost Cold Start Performance in Text Classification Eyal Shnarch∗, Ariel Gera∗, Alon Halfon∗, Lena Dankin, Leshem Choshen, Ranit Aharonov, Noam Slonim IBM Research {eyals, alonhal, lenad, leshem.choshen, noams}@il.ibm.com, ariel.gera1@ibm.com, ranitah1@gmail.com Abstract In real-world scenarios, a text classification task often begins with a cold start, when la- beled data is scarce. In such cases, the com- mon practice of fine-tuning pre-trained mod- els, such as BERT, for a target classification task, is prone to produce poor performance. We suggest a method to boost the performance of such models by adding an intermediate un- supervised classification task, between the pre- training and fine-tuning phases. As such an intermediate task, we perform clustering and train the pre-trained model on predicting the cluster labels. We test this hypothesis on var- ious data sets, and show that this additional classification phase can significantly improve performance, mainly for topical classification tasks, when the number of labeled instances available for fine-tuning is only a couple of dozen to a few hundred. 1 Introduction The standard paradigm for text classification relies on supervised learning, where it is well known that the size and quality of the labeled data strongly im- pact the performance (Raffel et al., 2019). Hence, developing a text classifier in practice typically re- quires making the most of a relatively small set of annotated examples. The emergence of transformer-based pre-trained language models such as BERT (Devlin et al., 2018) has reshaped the NLP landscape, leading to significant advances in the performance of most NLP tasks, text classification included (e.g., Nogueira and Cho, 2019; Ein-Dor et al., 2020). These models typically rely on pretraining with massive and heterogeneous corpora on a general Masked Language Modeling (MLM) task, i.e., pre- dicting a word that is masked in the original text. Later on, the obtained model is fine-tuned to the actual task of interest, termed here the target task, ∗These authors contributed equally to this work. using the labeled data available for this task. Thus, pretrained models serve as general sentence en- coders which can be adapted to a variety of target tasks (Lacroix et al., 2019; Wang et al., 2020a). Our work focuses on a challenging yet common scenario, where unlabeled data is available but la- beled data is scarce. In many real-world scenar- ios, obtaining even a couple of hundred of labeled examples per class is challenging. Commonly, a target class has a relatively low prior in the exam- ined data, making it a formidable goal to collect enough positive examples for it (Japkowicz and Stephen, 2002). Moreover, sometimes data cannot be labeled via crowd-annotation platforms due to its confidentiality (be it for data privacy reasons or for protecting intellectual property) or since the labeling task requires special expertise. On top of this, often the number of categories to be consid- ered is relatively large, …

Content truncated to 500 words