MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2523–2548 May 2-6, 2023 ©2023 Association for Computational Linguistics MAPL : Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting Oscar Mañas△ Pau Rodriguez∗,♢ Saba Ahmadi∗,△ Aida Nematzadeh♡ Yash Goyal♣ Aishwarya Agrawal△,† △Mila, Université de Montréal ♢ServiceNow Research ♡DeepMind ♣Samsung - SAIT AI Lab, Montreal †Canada CIFAR AI Chair oscar.manas@mila.quebec Abstract Large pre-trained models have proved to be remarkable zero- and (prompt-based) few- shot learners in unimodal vision and language tasks. We propose MAPL, a simple and parameter-efficient method that reuses frozen pre-trained unimodal models and leverages their strong generalization capabilities in mul- timodal vision-language (VL) settings. MAPL learns a lightweight mapping between the rep- resentation spaces of unimodal models using aligned image-text data, and can generalize to unseen VL tasks from just a few in-context examples. The small number of trainable pa- rameters makes MAPL effective at low-data and in-domain learning. Moreover, MAPL’s modularity enables easy extension to other pre- trained models. Extensive experiments on sev- eral visual question answering and image cap- tioning benchmarks show that MAPL achieves superior or competitive performance compared to similar methods while training orders of magnitude fewer parameters. MAPL can be trained in just a few hours using modest com- putational resources and public datasets. We release our code and pre-trained model weights at https://github.com/oscmansan/mapl. 1 Introduction Over the past few years, natural language process- ing and computer vision have witnessed impressive progress in learning models capable of transferring to unseen tasks or benchmarks (Brown et al., 2020; Zhang et al., 2022; Radford et al., 2021; Jia et al., 2021). Recently referred to as foundation mod- els (Bommasani et al., 2021), these can be adapted to a wide range of unimodal vision and language tasks without any additional training. In this work, we study reusing such powerful unimodal foundation models for multimodal vision- language (VL) downstream tasks. In particular, we propose to connect a vision encoder, such as *denotes equal contribution. CLIP (Radford et al., 2021), to an autoregressive language model (LM), such as GPT (Radford et al., 2018, 2019; Brown et al., 2020), with minimal additional training on multimodal data. Our goal is to obtain a single VL model that can leverage the in-context learning abilities (Brown et al., 2020) of the pre-trained LM to generalize to unseen VL tasks from just a few examples. One challenge in connecting vision encoders with LMs is aligning the visual and textual rep- resentation spaces. Recent works have approached this by adapting the LM to visual representations, either by fine-tuning the entire LM (Dai et al., 2022) or training adapter layers (Eichenberg et al., 2021; Alayrac et al., 2022). These systems are computa- tionally expensive to train as they have a large num- ber of learnable parameters (hundreds of millions to a few billions) and use large-scale multimodal train- ing data. On the other hand, Frozen (Tsimpoukelli et al., 2021) keeps the LM frozen, thus learning ∼10× less …

Content truncated to 500 words