How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input?

Proceedings of the 29th International Conference on Computational Linguistics, pages 5582–5596 October 12–17, 2022. 5582 How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input? Lovisa Hagström1 Richard Johansson1,2 1Chalmers University of Technology, 2University of Gothenburg {lovhag, richajo}@chalmers.se Abstract Current language models have been criticised for learning language from text alone without connection between words and their meaning. Consequently, multimodal training has been proposed as a way for creating models with better language understanding by providing the lacking connection. We focus on pre-trained multimodal vision-and-language (VL) models for which there already are some results on their language understanding capabilities. An unresolved issue with evaluating the linguistic skills of these models, however, is that there is no established method for adapting them to text-only input without out-of-distribution un- certainty. To find the best approach, we investi- gate and compare seven possible methods for adapting three different pre-trained VL mod- els to text-only input. Our evaluations on both GLUE and Visual Property Norms (VPN) show that care should be put into adapting VL mod- els to zero-shot text-only tasks, while the mod- els are less sensitive to how we adapt them to non-zero-shot tasks. We also find that the adap- tation methods perform differently for different models and that unimodal model counterparts perform on par with the VL models regard- less of adaptation, indicating that current VL models do not necessarily gain better language understanding from their multimodal training. 1 Introduction Having models learn language from text alone has been criticised based on several aspects, from fun- damental arguments about how language works (Bender and Koller, 2020) to findings on lack of certain information in text (Gordon and Van Durme, 2013; Paik et al., 2021). To train language mod- els on more sources than text is therefore a pro- posed direction for creating language models with better language understanding (Bisk et al., 2020). These models would then become multimodal, with the capability to process both text and information from other modalities. Figure 1: An overview of different ways to adapt a multimodal model to text-only input. It showcases three of the seven adaptations evaluated in this work. The multimodal models of interest in this work are vision-and-language (VL) models that have been trained on images and their corresponding captions or visual questions (Lu et al., 2019; Tan and Bansal, 2019; Su et al., 2020; Li et al., 2019; Chen et al., 2020). These models are performant on several image-text tasks such as image captioning and VQA, while there also is an increased interest for evaluating how their natural language under- standing is influenced by their multimodal training (Iki and Aizawa, 2021; Yun et al., 2021). It is however tricky to investigate the pure natural language understanding of the aforementioned VL models, since their language processing is condi- tioned on visual features. For certain investigations, we may simply wish to evaluate the models on text- only domains, while these models have not been developed for this purpose. If we do not attend to …

Content truncated to 500 words