CPL: Counterfactual Prompt Learning for Vision and Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3407 - 3418 December 7-11, 2022 ©2022 Association for Computational Linguistics CPL: Counterfactual Prompt Learning for Vision and Language Models Xuehai He1 Diji Yang1 Weixi Feng2 Tsu-Jui Fu2 Arjun Akula3 Varun Jampani3 Pradyumna Narayana3 Sugato Basu3 William Yang Wang2 Xin Eric Wang1 1UC Santa Cruz, 2UC Santa Barbara, 3Google {xhe89,dyang39,xwang366}@ucsc.edu {weixifeng,tsu-juifu,william}@ucsb.edu {arjunakula,varunjampani,pradyn,sugato}@google.com Abstract Prompt tuning is a new few-shot transfer learning technique that only tunes the learn- able prompt for pre-trained vision and lan- guage models such as CLIP. However, ex- isting prompt tuning methods tend to learn spurious or entangled representations, which leads to poor generalization to unseen con- cepts. Towards non-spurious and efficient prompt learning from limited examples, this paper presents a novel Counterfactual Prompt Learning (CPL) method for vision and lan- guage models, which simultaneously employs counterfactual generation and contrastive learn- ing in a joint optimization framework. Partic- ularly, CPL constructs counterfactual by iden- tifying minimal non-spurious feature change between semantically-similar positive and neg- ative samples that causes concept change and learns more generalizable prompt representa- tion from both factual and counterfactual exam- ples via contrastive learning. Extensive exper- iments demonstrate that CPL can obtain supe- rior few-shot performance on different vision and language tasks than previous prompt tuning methods on CLIP. On image classification, we achieve a 3.55% average relative improvement on unseen classes across seven datasets; on image-text retrieval and visual question answer- ing, we gain up to 4.09% and 25.08% relative improvements across three few-shot scenarios on unseen test sets respectively.1 1 Introduction Pre-trained vision and language foundation mod- els (Radford et al., 2021; Jia et al., 2021) have shown encouraging results toward open-domain visual-concept matching. Benefiting from prompt engineering (Song et al., 2022a; Liu et al., 2022), where free-form text prompts are designed for spe- cific task goals, those foundation models can be easily transferred to a wide array of tasks under 1Our code is released at https://github.com/ eric-ai-lab/CPL. A: A large long train on a steel track B: A large long train on a steel track near a barn What if we add a barn to image A (or remove the barn from image B)? Will the prompt be changed ? Figure 1: A conceptual overview of counterfactual prompt learning. CPL constructs counterfactuals by identifying non-spurious feature change that causally causes the prompt change. In this case, the “barn” fea- ture is the essential cause between Prompt A and B. zero-shot and few-shot scenarios, including im- age classification (Deng et al., 2009), visual ques- tion answering (Shen et al., 2021), image-text re- trieval (Jia et al., 2021), etc. But manually con- structing prompts for vision and language models such as CLIP is a tedious, time-consuming process, which usually requires prior domain knowledge and leads to suboptimal solutions. Prompt tuning (Lester et al., 2021), on the other hand, liberates us from manual prompt engineering and automates this process. Prompt tuning meth- ods (Ju et al., 2021; Lin et al., 2014; …

Content truncated to 500 words