A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2763 - 2775 May 22-27, 2022 c⃝2022 Association for Computational Linguistics A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models Woojeong Jin1∗ Yu Cheng2 Yelong Shen2 Weizhu Chen2 Xiang Ren1 1University of Southern California 2Microsoft Corporation {woojeong.jin,xiangren}@usc.edu {yu.cheng,yelong.shen,wzchen}@microsoft.com Abstract Large pre-trained vision-language (VL) mod- els can learn a new task with a handful of examples and generalize to a new task with- out fine-tuning. However, these VL mod- els are hard to deploy for real-world applica- tions due to their impractically huge sizes and slow inference speed. To solve this limitation, we study prompt-based low-resource learn- ing of VL tasks with our proposed method, FEWVLM, relatively smaller than recent few- shot learners. For FEWVLM, we pre-train a sequence-to-sequence transformer model with prefix language modeling (PrefixLM) and masked language modeling (MaskedLM). Fur- thermore, we analyze the effect of diverse prompts for few-shot tasks. Experimental results on VQA show that FEWVLM with prompt-based learning outperforms Frozen (Tsimpoukelli et al., 2021) which is 31× larger than FEWVLM by 18.2% point and achieves comparable results to a 246× larger model, PICa (Yang et al., 2021). In our analysis, we observe that (1) prompts significantly af- fect zero-shot performance but marginally af- fect few-shot performance, (2) models with noisy prompts learn as quickly as hand-crafted prompts given larger training data, and (3) MaskedLM helps VQA tasks while PrefixLM boosts captioning performance. Our code is publicly available at https://github. com/woojeongjin/FewVLM 1 Introduction Fine-tuning large pre-trained language models (PLMs) have led to strong results in various do- mains including vision-language tasks (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020; Radford et al., 2021). Such large PLMs can learn a new task with a few examples or generalize to a new task without fine-tuning on any train- ing examples, i.e., few-shot and zero-shot learn- ∗ Work was mainly done while interning at Microsoft Azure AI. question: What position is this man playing? answer: <text_1> an image of a small black dog standing over a plate of food. VQA Captioning Input image Input text <text_1> pitcher Target text Input text Target text Input image Figure 1: Examples of VQA and Captioning tasks. In our setup, we convert the tasks into generative tasks in which models need to generate target text given input text and an image. ing (Brown et al., 2020; Radford et al., 2021; Tsim- poukelli et al., 2021). Few-shot learning overcomes the challenges of data-hungry supervised learning, where collecting human-labeled data is costly and slow. However, recent few-shot models such as GPT3 (Brown et al., 2020), Frozen (Tsimpoukelli et al., 2021), and PICa (Yang et al., 2021) are too large to deploy in small or moderate computing machines due to their gigantic model sizes In this paper, we study low-resource learning of VL tasks with our proposed method, FEWVLM, a moderate-sized vision-language model, in which we fine-tune the model with no …

Content truncated to 500 words