PESCO: Prompt-enhanced Self Contrastive Learning for Zero-shot Text Classification

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14897–14911 July 9-14, 2023 ©2023 Association for Computational Linguistics PESCO: Prompt-enhanced Self Contrastive Learning for Zero-shot Text Classification Yau-Shian Wang Ta-Chung Chi Ruohong Zhang Yiming Yang Carnegie Mellon University king6101@gmail.com {tachungc,ruohongz}@andrew.cmu.edu yiming@cs.cmu.edu Abstract We present PESCO, a novel contrastive learn- ing framework that substantially improves the performance of zero-shot text classification. We formulate text classification as a neural text matching problem where each document is treated as a query, and the system learns the mapping from each query to the relevant class labels by (1) adding prompts to enhance la- bel matching, and (2) using retrieved labels to enrich the training set in a self-training loop of contrastive learning. PESCO achieves state-of-the-art performance on four benchmark text classification datasets. On DBpedia, we achieve 98.5% accuracy without any labeled data, which is close to the fully-supervised re- sult. Extensive experiments and analyses show all the components of PESCO are necessary for improving the performance of zero-shot text classification. 1 Introduction Text classification is the task of assigning relevant category labels to each input document. It is an im- portant problem in machine learning research with a wide spectrum of applications, including senti- ment analysis (Pang et al., 2002; Maas et al., 2011; Socher et al., 2013; Tang et al., 2014), question an- swering (Rajpurkar et al., 2016, 2018), and intent classification (Tur et al., 2010), etc. Recently, deep neural networks have obtained remarkable improve- ments in text classification, including CNNs (Kim, 2014; Zhang et al., 2015), RNNs (Tang et al., 2015; Yang et al., 2016), Transformers (Vaswani et al., 2017), and more, thanks to the successful modeling of contextualized representations. Despite the remarkable progress, training well- performing neural classifiers still requires a large amount of human-labeled documents, which is costly and time-consuming, especially for new application domains. This stimulates the recent trend of exploring self-supervised pre-training neu- ral models on text classification tasks. In particu- lar, pre-trained language models (PTLMs) (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019) clearly stand out from other methods owing to the pre-training on large-scale unlabeled data. Never- theless, how to adapt PTLMs to downstream tasks with less supervision remains an open question for the research community, inviting new ideas to ex- plore. Prompt-based learning (Brown et al., 2020; Shin et al., 2020; Liu et al., 2021; Li and Liang, 2021; Gao et al., 2021a) has been actively studied to bet- ter adapt PTLMs to downstream tasks with the goal of reducing human annotation effort. For example, PET (Schick and Schütze, 2020) is a prompt-based method for few-shot text classification. It formu- lates the task as a Cloze Test, where a PTLM is used to predict the output label(s) by completing a prompt concatenated right after an input document. For example, the sentiment of a product review is highly likely to be positive if a PTLM fills the word “good” into the following input: [Review] …

Content truncated to 500 words