CARETS: A Consistency And Robustness Evaluative Test Suite for VQA

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6392 - 6405 May 22-27, 2022 c⃝2022 Association for Computational Linguistics CARETS: A Consistency And Robustness Evaluative Test Suite for VQA Carlos E. Jimenez Olga Russakovsky Karthik Narasimhan Princeton University {carlosej,olgarus,karthikn}@princeton.edu Abstract We introduce CARETS, a systematic test suite to measure consistency and robustness of modern VQA models through a series of six fine-grained capability tests. In con- trast to existing VQA test sets, CARETS fea- tures balanced question generation to create pairs of instances to test models, with each pair focusing on a specific capability such as rephrasing, logical symmetry or image obfus- cation. We evaluate six modern VQA systems on CARETS and identify several actionable weaknesses in model comprehension, espe- cially with concepts such as negation, disjunc- tion, or hypernym invariance. Interestingly, even the most sophisticated models are sensi- tive to aspects such as swapping the order of terms in a conjunction or changing the number of answer choices mentioned in the question. We release CARETS to be used as an exten- sible tool for evaluating multi-modal model robustness.1 1 Introduction The task of visual question answering integrates the domains of computer vision and NLP by prob- ing models’ understanding of images through nat- ural language queries. After the introduction of the Visual Question Answering (VQA) bench- mark (Antol et al., 2015), subsequent work iden- tified the presence of several superficial correla- tions and other weaknesses latent in the VQA question gathering process (Goyal et al., 2017; Agrawal et al., 2018), which lead to potentially optimistic evaluations when considering accu- racy alone. More recently developed benchmarks (Hudson and Manning, 2019; Goyal et al., 2017; Agrawal et al., 2018) explicitly avoid these weak- nesses by introducing question, answer, and im- age balancing, or distributional shifts. While these 1Source code, data, and additional resources may be found at https://github.com/princeton-nlp/CARETS efforts provide more difficult benchmarks, a thor- ough evaluation of model capabilities requires a deeper and more detailed approach. To this end, we introduce CARETS – a Con- sistency And Robustness Evaluative Test Suite for visual question answering. Inspired by re- cent work in NLP that generates ‘unit tests’ for models (Ribeiro et al., 2020b), CARETS contains systematically generated VQA tests that evalu- ate six different capabilities that any VQA model should handle – robustness to question rephras- ings, ontological reasoning, symmetrical logic, visual perturbations, question negation, and at- tribute antonymy. Each test point in CARETS consists of a pair of instances which are small but strategic variations of each other either visually or in the question’s text. This allows us to con- duct fine-grained capability evaluations beyond just measuring high-level accuracy scores. Across tests, we generate over 190,000 instance pairs in total using a programmatic approach that fills in templates (from nearly 200 templates in total) using ground-truth scene graphs (Krishna et al., 2017) from the GQA (Hudson and Man- ning, 2019) validation split. We then evaluate six modern VQA models on each …

Content truncated to 500 words