CrossVQA: Scalably Generating Benchmarks for Systematically Testing VQA Generalization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2148–2166 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2148 CrossVQA: Scalably Generating Benchmarks For Systematically Testing VQA Generalization Arjun R. Akula1∗, Soravit Changpinyo3, Boqing Gong3, Piyush Sharma3, Song-Chun Zhu2,4,5, Radu Soricut3 1UCLA Center for Vision, Cognition, Learning, and Autonomy, 2Beijing Institute for General Artificial Intelligence (BIGAI), 3Google Research, 4Tsinghua University, 5Peking University aakula@ucla.edu, {schangpi,bgong,piyushsharma}@google.com, s.c.zhu@pku.edu.cn, rsoricut@google.com Abstract One challenge in evaluating visual question answering (VQA) models in the cross-dataset adaptation setting is that the distribution shifts are multi-modal, making it difficult to iden- tify if it is the shifts in visual or language features that play a key role. In this paper, we propose a semi-automatic framework for generating disentangled shifts by introducing a controllable visual question-answer genera- tion (VQAG) module that is capable of gen- erating highly-relevant and diverse question- answer pairs with the desired dataset style. We use it to create CrossVQA, a collection of test splits for assessing VQA generalization based on the VQA2, VizWiz, and Open Im- ages datasets. We provide an analysis of our generated datasets and demonstrate its utility by using them to evaluate several state-of-the- art VQA systems. One important finding is that the visual shifts in cross-dataset VQA matter more than the language shifts. More broadly, we present a scalable framework for systematically evaluating the machine with lit- tle human intervention. 1 Introduction Multiple datasets have been proposed to mea- sure the progress on visual question answering (VQA) (Antol et al., 2015; Zhu et al., 2016b; Goyal et al., 2017; Gurari et al., 2018; Hudson and Man- ning, 2019; Yang et al., 2016; Tu et al., 2014; Qi et al., 2015; Liu et al., 2016). However, these datasets often possess biases introduced in the data collection process and by the human annotators. It has been shown that existing VQA models lever- age these spurious biases and take shortcuts (Goyal et al., 2017; Agrawal et al., 2018; Chao et al., 2018a; Akula et al., 2020a). As a result, the perfor- mance of those models on a specific VQA dataset can only serve as a rough proxy for the true learning of the VQA task (Bras et al., 2020). ∗Work done in part while AA was an intern at Google. Test sets QAvqa2 QAvzwz Ivqa2 ✓  Ivzwz  ✓ Ioid   (a) Test sets QAvqa2 QAvzwz Ivqa2 ✓ ✓ Ivzwz ✓ ✓ Ioid ✓ ✓ (b) Table 1: (a) Existing VQA test sets; (b) CrossVQA (disen- tangled) test sets generated by our VQAG model. One common remedy to this is to go beyond in- domain evaluation, in which the test set exhibits some form of “distribution shifts” from the train- ing set (Agrawal et al., 2018; Chao et al., 2018b). The key idea is that a generalizable VQA model should be able to extrapolate, for example, from one dataset to another. One challenge that is quite unique to VQA in this setting is that the distribution shift is …

Content truncated to 500 words