Modular and Parameter-Efficient Multimodal Fusion with Prompting

Findings of the Association for Computational Linguistics: ACL 2022, pages 2976 - 2985 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Modular and Parameter-Efficient Multimodal Fusion with Prompting Sheng Liang, Mengjie Zhao, Hinrich Sch¨utze Center for Information and Language Processing (CIS) LMU Munich, Germany {shengliang,mzhao}@cis.lmu.de Abstract Recent research has made impressive progress in large-scale multimodal pre-training. In the context of the rapid growth of model size, it is necessary to seek efficient and flexible methods other than finetuning. In this paper, we propose to use prompt vectors to align the modalities. Our method achieves comparable performance to several other multimodal fusion methods in low-resource settings. We further show that our method is modular and parameter-efficient for processing tasks involving two or more data modalities. 1 Introduction The success of large-scale pretrained language models (PLMs; Devlin et al. (2019); Yang et al. (2019); Brown et al. (2020); Raffel et al. (2020)) and image encoders (Dosovitskiy et al., 2021; Liu et al., 2021b) has stimulated a surge of pretrained multimodal models (Lu et al., 2019; Tan and Bansal, 2019; Radford et al., 2021; Lin et al., 2021) that align text with data in other modalities. The fast-growing number of parameters in the pretrained models encourages researchers to create more data- and parameter-efficient methods than finetuning (Houlsby et al., 2019; Zhao et al., 2020; Zaken et al., 2021; Li and Liang, 2021; He et al., 2022). Recently, prompting – concatenating manu- ally designed prompt phrases (Schick and Sch¨utze, 2021; Tam et al., 2021; Le Scao and Rush, 2021; Zhao and Sch¨utze, 2021) or trained embedding vec- tors (Li and Liang, 2021; Lester et al., 2021) to the text input of PLMs – has become an important research direction. Following this trend, Tsimpoukelli et al. (2021) introduce Frozen, successfully extending PLMs into few-shot learners (i.e., models that perform well with only a handful of data) for multimodal tasks, by pretraining a vision encoder whose out- puts are prompts fed to the PLM. Frozen performs VE LM Embedding layer Language Model how many cats ? <P1> <P2> two Figure 1: Model architecture. We disentangle VE’s functionality by introducing prompt vectors. The only work of VE is to extract image representations. PLM and VE are fixed (grey) during training; two prompt vectors are the only trainable parameters (red). strongly on low-resource visual question answer- ing through GPT3-style (Brown et al., 2020) prim- ing (in-context learning). Frozen consists of two components: A vision encoder (VE) (in their case, NF-ResNet-50 (Brock et al., 2021)) and an off-the- shelf PLM like GPT3. When pretraining Frozen, the PLM takes the image representations extracted by VE as prompts, to generate captions describing the input image. PLM parameters are fixed and VE is pretrained from scratch. The success of Frozen shows the potential of prompting-based systems for solving multimodal tasks (Zhou et al., 2021; Yang et al., 2021; Salaberria et al., 2021). One inherent discrepancy between Frozen and prompting for NLP tasks (Li and Liang, 2021; Lester et al., 2021) is that …

Content truncated to 500 words