SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3103–3111 June 6–11, 2021. ©2021 Association for Computational Linguistics 3103 SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency Sameer Dharur1 Purva Tendulkar1→3 Dhruv Batra1,2 Devi Parikh1,2 Ramprasaath R. Selvaraju1→4 {sameerdharur, purva, dbatra, parikh, ramprs}@gatech.edu 1Georgia Tech 2Facebook AI Research 3University of California, San Diego 4Salesforce Research Abstract Recent research in Visual Question Answer- ing (VQA) has revealed state-of-the-art mod- els to be inconsistent in their understanding of the world – they answer seemingly difficult questions requiring reasoning correctly but get simpler associated sub-questions wrong. These sub-questions pertain to lower level vi- sual concepts in the image that models ide- ally should understand to be able to answer the reasoning question correctly. To address this, we first present a gradient-based inter- pretability approach to determine the ques- tions most strongly correlated with the rea- soning question on an image, and use this to evaluate VQA models on their ability to iden- tify the relevant sub-questions needed to an- swer a reasoning question. Next, we propose a contrastive gradient learning based approach called Sub-question Oriented Tuning (SOrT) which encourages models to rank relevant sub- questions higher than irrelevant questions for an <image, reasoning-question> pair. We show that SOrT improves model consistency by up to 6.5% points over existing approaches, while also improving visual grounding and ro- bustness to rephrasings of questions. 1 Introduction Current visual question answering (VQA) models struggle with consistency. They often correctly answer complex reasoning questions, i.e, those re- quiring common sense knowledge and logic on top of perceptual capabilities, but fail on associated low-level perception questions, i.e., those directly related to the visual content in the image. For e.g., in Fig 1, models answer the reasoning question “Was this taken in the daytime?” correctly, but fail on the associated perception question “Is the sky bright?” indicating that the models likely answered the reasoning question correctly for the wrong rea- son(s). In this work, we explore the usefulness of leveraging information about sub-questions, i.e., low-level perception questions relevant to a rea- soning question, and irrelevant questions, i.e., any other questions about the image unrelated to the rea- soning question, to improve consistency in VQA. Selvaraju et al. (2020) have studied this problem and introduced the VQA-Introspect dataset that draws a distinction between higher-level reasoning questions and lower-level perception sub-questions. We augment this dataset with additional perception questions from the VQAv2 dataset such that each <image, reasoning question> pair contains a set of relevant perception questions, which we refer to as sub-questions (e.g.,“Is the sky bright?” in Fig 1) and irrelevant perception questions, which we refer to as irrelevant questions (e.g., “Is the train moving?” in Fig 1) throughout this paper. We use Gradient-based Class Activation Map- ping (Grad-CAM) vectors (Selvaraju et al., 2019a) – a faithful function of the model’s parameters, question, answer and image – to propose an inter- pretability technique that determines the questions most …

Content truncated to 500 words