A negative case analysis of visual grounding methods for VQA

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8172–8181 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8172 A Negative Case Analysis of Visual Grounding Methods for VQA Robik Shrestha1 Kushal Kafle1,2 Christopher Kanan1,3,4 Rochester Institute of Technology1 Adobe Research2 Paige3 Cornell Tech4 {rss9369, kk6055, kanan}@rit.edu Abstract Existing Visual Question Answering (VQA) methods tend to exploit dataset biases and spu- rious statistical correlations, instead of produc- ing right answers for the right reasons. To ad- dress this issue, recent bias mitigation meth- ods for VQA propose to incorporate visual cues (e.g., human attention maps) to better ground the VQA models, showcasing impres- sive gains. However, we show that the perfor- mance improvements are not a result of im- proved visual grounding, but a regularization effect which prevents over-fitting to linguis- tic priors. For instance, we find that it is not actually necessary to provide proper, human- based cues; random, insensible cues also re- sult in similar improvements. Based on this observation, we propose a simpler regulariza- tion scheme that does not require any external annotations and yet achieves near state-of-the- art performance on VQA-CPv21. 1 Introduction Visual Question Answering (VQA) (Antol et al., 2015), the task of answering questions about visual content, was proposed to facilitate the development of models with human-like visual and linguistic understanding. However, existing VQA models often exploit superficial statistical biases to produce responses, instead of producing the right answers for the right reasons (Kafle et al., 2019). The VQA-CP dataset (Agrawal et al., 2018) showcases this phenomenon by incorporating dif- ferent question type/answer distributions in the train and test sets. Since the linguistic priors in the train and test sets differ, models that exploit these priors fail on the test set. To tackle this issue, recent works have endeavored to enforce proper visual grounding, where the goal is to make models produce answers by looking at relevant visual regions (Gan et al., 2017; Selvaraju et al., Answer distribution VQA-CP Dataset Prediction: Brown Baseline Methods Affected by language priors Green Brown Q: What color is the couch? A: Green Training Test Green Brown Fail to generalize Prediction: Green Recent Methods Improve by grounding on relevant regions +9% over baselines Prediction: Green Our Findings Irrelevant/random regions result in similar gains +9% over baselines Figure 1: We find that existing visual sensitivity en- hancement methods improve performance on VQA- CPv2 through regularization as opposed to proper vi- sual grounding. 2019; Wu and Mooney, 2019), instead of exploit- ing linguistic priors. These approaches rely on additional annotations/cues such as human-based attention maps (Das et al., 2017), textual expla- nations (Huk Park et al., 2018) and object label predictions (Ren et al., 2015) to identify relevant regions, and train the model to base its predictions on those regions, showing large improvements (8- 10% accuracy) on the VQA-CPv2 dataset. Here, we study these methods. We find that their improved accuracy does not actually emerge from proper visual grounding, but from regularization effects, where the …

Content truncated to 500 words