CLIP Models are Few-Shot Learners: Empirical Studies on VQA and Visual Entailment

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6088 - 6100 May 22-27, 2022 c⃝2022 Association for Computational Linguistics CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment Haoyu Song†∗, Li Dong‡, Wei-Nan Zhang†, Ting Liu†, Furu Wei‡ † Harbin Institute of Technology ‡ Microsoft Research {hysong,wnzhang,tliu}@ir.hit.edu.cn {lidong1,fuwei}@microsoft.com Abstract CLIP has shown a remarkable zero-shot capa- bility on a wide range of vision tasks. Pre- viously, CLIP is only regarded as a power- ful visual encoder. However, after being pre- trained by language supervision from a large amount of image-caption pairs, CLIP itself should also have acquired some few-shot abil- ities for vision-language tasks. In this work, we empirically show that CLIP can be a strong vision-language few-shot learner by leverag- ing the power of language. We first evaluate CLIP’s zero-shot performance on a typical vi- sual question answering task and demonstrate a zero-shot cross-modality transfer capability of CLIP on the visual entailment task. Then we propose a parameter-efficient fine-tuning strategy to boost the few-shot performance on the vqa task. We achieve competitive zero/few- shot results on the visual question answering and visual entailment tasks without introduc- ing any additional pre-training procedure. 1 Introduction Vision-language understanding (VLU) tasks, such as visual question answering (Antol et al., 2015) and visual entailment (Xie et al., 2019), test a sys- tem’s ability to comprehensively understand the se- mantics of both visual world and natural language. To capture the alignment between vision and lan- guage, various efforts have been made to build the vision-language pre-trained models (Lu et al., 2019; Chen et al., 2020; Su et al., 2020; Zhang et al., 2021; Wang et al., 2021). Despite their supe- rior performances, these methods have extensively utilized human-annotated training data that are ex- pensive or require expert knowledge, such as object detection datasets (Lin et al., 2014; Kuznetsova et al., 2020) and aligned image-text pairs (Deng et al., 2009; Sharma et al., 2018). Collecting such datasets requires heavy work on data gathering and ∗Contribution during internship at Microsoft Research. Figure 1: Examples of the two vision-language under- standing tasks. For VQA, language prompts are used. For visual entailment, caption and hypothesis, i.e., text- text pairs, are used in training, while image and hypoth- esis, i.e., image-text pairs, are used at inference. human annotation, and thus their scales are only in the realm of tens of millions, which are much smaller than the Internet text corpora for NLP pre- training (Devlin et al., 2019; Brown et al., 2020). Recently, CLIP (Radford et al., 2021) has been proposed to learn visual concepts with natural lan- guage supervision, where its 400 million image- text pairs are crawled from the Internet. CLIP consists of a visual encoder and a text encoder, and it learns visual representations by aligning im- ages and texts through contrastive loss. In this way, CLIP achieves strong zero-shot performances on vi- sion benchmarks such as ImageNet. Besides, Shen et al. (2022) prove that …

Content truncated to 500 words