Fine-tuning vs From Scratch: Do Vision & Language Models Have Similar Capabilities on Out-of-Distribution Visual Question Answering?

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1496–1508 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1496 Fine-tuning vs From Scratch: Do Vision & Language Models Have Similar Capabilities on Out-of-Distribution Visual Question Answering? Kristian Nørgaard Jensen♦ Barbara Plank♦♣ ♦Department of Computer Science, IT University of Copenhagen, Denmark ♣Center for Information and Language Processing (CIS), LMU Munich, Germany krnj@itu.dk bplank@cis.uni-muenchen.de Abstract Fine-tuning general-purpose pre-trained models has become a de-facto standard, also for Vision and Language tasks such as Visual Question Answering (VQA). In this paper, we take a step back and ask whether a fine-tuned model has superior linguistic and reasoning capabilities than a prior state-of-the-art architecture trained from scratch on the training data alone. We perform a fine-grained evaluation on out-of-distribution data, including an analysis on robustness due to linguistic variation (rephrasings). Our empirical results confirm the benefit of pre-training on overall performance and rephrasing in particular. But our results also uncover surprising limitations, particularly for answering questions involving boolean operations. To complement the empirical evaluation, this paper also surveys relevant earlier work on 1) available VQA data sets, 2) models developed for VQA, 3) pre-trained Vision+Language models, and 4) earlier fine-grained evaluation of pre-trained Vision+Language models. Keywords: Visual Question Answering, Multimodal Transformers, Vision+Language, Fine-grained Evaluation 1. Introduction Large general-purpose pre-trained machine learning models have become ubiquitous. While new mod- els emerge at a rapid pace, only limited research ex- ists into the robustness of such models (Li et al., 2020b; Cao et al., 2020). In particular, it remains an open question whether pre-training is the best way for Language+Vision tasks. With a growing number of general-purpose pre-trained Vision+Language (V+L) models, which all suggest that they are better than the previous one, it is important to take a step back, and investigate and analyse models. In this paper, we perform a fine-grained evaluation of two models on Visual Question Answering (VQA). We compare a fine-tuned general-purpose V+L model to an earlier task-specific model trained from scratch on the standard VQA v2 data set (Goyal et al., 2017). We examine their linguistic and reasoning capabilities, as well as their robustness to language variation on three out-of-distribution data sets in a zero-shot fash- ion. Moreover, we also review early Visual Question Answering models, VQA data sets, recent pre-trained V+L models and survey related work on their evalua- tion. Our analysis focuses on BAN (Kim et al., 2018) and LXMERT (Tan and Bansal, 2019), each state-of- the-art VQA model at their respective times. Our re- sults show that their capabilities diverge substantially, and while fine-tuned LXMERT performs the best on most setups, it is not always the case. Strikingly, BAN does perform better on datasets involving boolean compositions (VQA-Compose and VQA-Supplement). LXMERT is particularly susceptible to questions in- volving OR operations or multiple boolean compar- isons in contrast to BAN, suggesting that even large pre-trained models perform insufficiently and answer VQA v2 (Train) BAN VQA-LOL LXMERT VQA v2 (Train) VQA v2 …

Content truncated to 500 words