Beyond prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8560 - 8579 December 7-11, 2022 ©2022 Association for Computational Linguistics Beyond prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations Yu Fei1, Zhao Meng∗1, Ping Nie∗2, Roger Wattenhofer1, Mrinmaya Sachan1 1ETH Zurich, 2Peking University feiyuwalter@gmail.com, {zhmeng, wattenhofer}@ethz.ch, ping.nie@pku.edu.cn, mrinmaya.sachan@inf.ethz.ch Abstract Recent work has demonstrated that pre-trained language models (PLMs) are zero-shot learn- ers. However, most existing zero-shot methods involve heavy human engineering or compli- cated self-training pipelines, hindering their ap- plication to new situations. In this work, we show that zero-shot text classification can be improved simply by clustering texts in the em- bedding spaces of PLMs. Specifically, we fit the unlabeled texts with a Bayesian Gaussian Mixture Model after initializing cluster posi- tions and shapes using class names. Despite its simplicity, this approach achieves superior or comparable performance on both topic and sentiment classification datasets and outper- forms prior works significantly on unbalanced datasets. We further explore the applicability of our clustering approach by evaluating it on 14 datasets with more diverse topics, text lengths, and numbers of classes. Our approach achieves an average of 20% absolute improvement over prompt-based zero-shot learning. Finally, we compare different PLM embedding spaces and find that texts are well-clustered by topics even if the PLM is not explicitly pre-trained to gen- erate meaningful sentence embeddings. This work indicates that PLM embeddings can cat- egorize texts without task-specific fine-tuning, thus providing a new way to analyze and utilize their knowledge and zero-shot learning ability1. 1 Introduction Recent developments in large pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020a) open up the possibil- ity of classifying texts without massive in-task data annotation. Such a zero-shot setting is receiving increasing attention as it is a good way to eval- uate the generalizability of knowledge in PLMs. *Equal contribution. 1Code and datasets available at: https://github.com/ fywalter/simptc Currently, most existing methods either utilize key- words for self-training (Chang et al., 2008; Meng et al., 2018; Wang et al., 2021) or reformulate the classification task into a cloze task using prompts (Brown et al., 2020; Schick and Schütze, 2021; Gao et al., 2021a). Keyword-based methods usually train multiple modules sequentially (Meng et al., 2020b), while prompting methods depend heavily on human engineering (Liu et al., 2021) or external knowledge (Hu et al., 2021). Such task-specific training or engineering is inefficient and usually does not generalize well to new applications. In this work, we show that we can better elicit the zero-shot text classification abilities of PLMs sim- ply by clustering texts in their embedding spaces. We draw inspiration from recent findings (Aha- roni and Goldberg, 2020) that texts in the same domain (e.g., legal or medical texts) tend to be clus- tered together in the PLM embedding spaces. This indicates that PLMs already have the knowledge to distinguish texts with different meanings. Fol- lowing this idea, we propose SimPTC: A Simple Probabilistic Text Classification framework …

Content truncated to 500 words