Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization

Findings of the Association for Computational Linguistics: EACL 2023, pages 1201–1226 May 2-6, 2023 ©2023 Association for Computational Linguistics Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization Aishwarya Agrawal∗,‡,♦ Ivana Kaji´c∗,♦ Emanuele Bugliarello∗,△ Elnaz Davoodi†,♦ Anita Gergely†,♦ Phil Blunsom♮ Aida Nematzadeh∗,‡,♦ ♦DeepMind △University of Copenhagen ♮University of Oxford {aiagrawal,kivana,nematzadeh}@deepmind.com emanuele@di.ku.dk Abstract Vision-and-language (V&L) models pre- trained on large-scale multimodal data have demonstrated strong performance on various tasks such as image captioning and visual question answering (VQA). The quality of such models is commonly assessed by measuring their performance on unseen data that typically comes from the same distribution as the training data. However, when evaluated under out-of-distribution (out-of-dataset) settings for VQA, we observe that these models exhibit poor generalization. We comprehensively evaluate two pretrained V&L models under different settings (i.e. classification and open-ended text generation) by conducting cross-dataset evaluations. We find that these models tend to learn to solve the benchmark, rather than learning the high-level skills required by the VQA task. We also find that in most cases generative models are less susceptible to shifts in data distribution compared to discriminative ones, and that multimodal pretraining is generally helpful for OOD generalization. Finally, we revisit assumptions underlying the use of automatic VQA evaluation metrics, and empirically show that their stringent nature repeatedly penalizes models for correct responses. 1 Introduction Visual Question Answering (VQA) is the task of automatically answering natural language open- ended questions about images. Tackling VQA in- volves multiple skills, such as language and visual understanding, integrating information between the two (vision and language) modalities, and com- monsense and knowledge based reasoning. One of the goals of the VQA research has been fostering the development of systems that are able to answer any open-ended question about any image. This ∗denotes equal first author contribution. † denotes equal contribution. ‡ denotes equal senior contribution. Detailed contributions follow at the end of the paper. motivation has inspired a fruitful line of research in designing VQA benchmarks (e.g., Malinowski and Fritz, 2014; Antol et al., 2015; Krishna et al., 2017; Goyal et al., 2017; Hudson and Manning, 2019) and models (e.g., Yang et al., 2015; Anderson et al., 2018; Lu et al., 2019; Cho et al., 2021). In this work, we investigate if recent pretrained VQA models can indeed answer any open-ended question about images or if they are mostly suitable for answering questions from the VQA benchmarks they are optimized for. In other words, are models learning to solve the task or learning to solve the datasets? We believe the former is more aligned with the goal of building real-world VQA systems. To measure whether models learn to solve the task of VQA, we believe we need to examine their out-of-distribution (OOD) generalization capabil- ities: how they perform on examples drawn from a distribution other than that of the training set. In this work, we extensively evaluate OOD gen- eralization of current pretrained V&L models by conducting cross-dataset evaluations (without any adaptation to the …

Content truncated to 500 words