Visual Prompt Tuning for Few-Shot Text Classification

Proceedings of the 29th International Conference on Computational Linguistics, pages 5560–5570 October 12–17, 2022. 5560 Visual Prompt Tuning for Few-Shot Text Classification Jingyuan Wen1,2, Yutian Luo1,2, Nanyi Fei1,2, Guoxing Yang1,2, Zhiwu Lu1,2,∗ Hao Jiang3, Jie Jiang3, Zhao Cao3 1Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China 2Beijing Key Laboratory of Big Data Management and Analysis Methods 3Huawei Poisson Lab, Hangzhou, Zhejiang {wenjingyuan,luzhiwu}@ruc.edu.cn Abstract Deploying large-scale pre-trained models in the prompt-tuning paradigm has demonstrated promising performance in few-shot learn- ing. Particularly, vision-language pre-training models (VL-PTMs) have been intensively ex- plored in various few-shot downstream tasks. However, most existing works only apply VL- PTMs to visual tasks like image classifica- tion, with few attempts being made on lan- guage tasks like text classification. In few- shot text classification, a feasible paradigm for deploying VL-PTMs is to align the input samples and their category names via the text encoders. However, it leads to the waste of visual information learned by the image en- coders of VL-PTMs. To overcome this draw- back, we propose a novel method named Vi- sual Prompt Tuning (VPT). To our best knowl- edge, this method is the first attempt to de- ploy VL-PTM in few-shot text classification task. The main idea is to generate the im- age embeddings w.r.t. category names as vi- sual prompt and then add them to the aligning process. Extensive experiments show that our VPT can achieve significant improvements un- der both zero-shot and few-shot settings. Im- portantly, our VPT even outperforms the most recent prompt-tuning methods on five public text classification datasets. 1 Introduction Pre-training models have achieved great success across a variety of tasks in recent years. Pre- training language models (PLMs) like BERT (De- vlin et al., 2019), GPT (Radford et al., 2018) and their variants (Liu et al., 2019; Raffel et al., 2020; Yang et al., 2019; Lewis et al., 2020) firstly ap- peared as the milestones in the AI field. They brought huge boost to natural language process- ing (NLP) tasks, such as text classification (Devlin et al., 2019), named entity recognition (NER) (Jia *Zhiwu Lu is the corresponding author. et al., 2020), and text generation (Chan and Fan, 2019). In computer vision, large-scale pre-training models (e.g., BiT (Kolesnikov et al., 2020) and ViT (Dosovitskiy et al., 2021)) became popular as in NLP. With convolutional neural networks or Transformers (Vaswani et al., 2017) as the backbones, they were shown to be effective on a wide range of visual downstream tasks (e.g., im- age classification, object detection, and semantic segmentation). More recently, inspired by these pre-training models in NLP and computer vision, vision-language pre-training models (VL-PTMs) have been intensively explored (Su et al., 2020; Li et al., 2020; Huo et al., 2021; Lu et al., 2022; Fei et al., 2022). They achieve excellent performance in cross-modal tasks like image-text retrieval, vi- sual question answering (VQA), and image cap- tion. Besides, they also show great potential in single-modal tasks (Lin et al., 2021; Yuan et al., 2021). These achievements …

Content truncated to 500 words