CELDA: Leveraging Black-box Language Model as Enhanced Classifier without Labels

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4364–4379 July 9-14, 2023 ©2023 Association for Computational Linguistics CELDA: Leveraging Black-box Language Model as Enhanced Classifier without Labels Hyunsoo Cho♢, Youna Kim♢, Sang-goo Lee♢♣ ♢Seoul National University, ♣IntelliSys {johyunsoo, anna9812, sglee}@europa.snu.ac.kr Abstract Utilizing language models (LMs) without inter- nal access is becoming an attractive paradigm in the field of NLP as many cutting-edge LMs are released through APIs and boast a mas- sive scale. The de-facto method in this type of black-box scenario is known as prompting, which has shown progressive performance en- hancements in situations where data labels are scarce or unavailable. Despite their efficacy, they still fall short in comparison to fully su- pervised counterparts and are generally brittle to slight modifications. In this paper, we pro- pose Clustering-Enhanced Linear Discrimina- tive Analysis (CELDA), a novel approach that improves the text classification accuracy with a very weak-supervision signal (i.e., name of the labels). Our framework draws a precise de- cision boundary without accessing weights or gradients of the LM model or data labels. The core ideas of CELDA are twofold: (1) extract- ing a refined pseudo-labeled dataset from an un- labeled dataset, and (2) training a lightweight and robust model on the top of LM, which learns an accurate decision boundary from an extracted noisy dataset. Throughout in-depth investigations on various datasets, we demon- strated that CELDA reaches new state-of-the- art in weakly-supervised text classification and narrows the gap with a fully-supervised model. Additionally, our proposed methodology can be applied universally to any LM and has the potential to scale to larger models, making it a more viable option for utilizing large LMs. 1 Introduction Large-scale language models (LMs) have been a driving force behind a series of breakthroughs in the machine-learning community. Despite their pre- eminence in wide applications, large LMs are often costly or infeasible to fine-tune as many distributed large models, such as GPT-3 (Brown et al., 2020), are provided in a black-box manner, which only allows limited access through commercial APIs. 60 70 80 90 100 IMDb (2) SST2 (2) AGNews (4) Yahoo (10) KPT CELDA LDA (full) Figure 1: Text classification accuracy on 4 benchmark datasets on T5 (11B). CELDA significantly improves the performance without model adaptation or labeled data closing the gap with fully labeled methods. To circumvent the explicit adaptation of the mod- els, many recent research leverage prompting, a training-free approach that elicits desired predic- tions from LMs by curating input into a more con- ceivable form. By doing so, prompting has shown remarkable improvements in data-scarce scenarios (i.e., zero-shot, few-shot), reminiscing the potential of large LMs as a universal, off-the-shelf solution for diverse tasks. Yet, it is still premature as their performance lags far behind the fine-tuned model and displays fragility to negligible changes (Lu et al., 2022; Perez et al., 2021). In this paper, we aim to bridge this gap by uti- lizing an unlabeled dataset without adapting the model …

Content truncated to 500 words