Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5860–5865, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5860 Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation 1Arijit Ray, 1Karan Sikka, 1Ajay Divakaran, *Stefan Lee, 1Giedrius Burachas 1first.last@sri.com, *steflee@gatech.edu 1SRI International, *Georgia Institute of Technology Abstract While models for Visual Question Answering (VQA) have steadily improved over the years, interacting with one quickly reveals that these models lack consistency. For instance, if a model answers “red” to “What color is the bal- loon?”, it might answer “no” if asked, “Is the balloon red?”. These responses violate sim- ple notions of entailment and raise questions about how effectively VQA models ground language. In this work, we introduce a dataset, ConVQA, and metrics that enable quantita- tive evaluation of consistency in VQA. For a given observable fact in an image (e.g. the bal- loon’s color), we generate a set of logically consistent question-answer (QA) pairs (e.g. Is the balloon red?) and also collect a human- annotated set of common-sense based consis- tent QA pairs (e.g. Is the balloon the same color as tomato sauce?). Further, we pro- pose a consistency-improving data augmen- tation module, a Consistency Teacher Mod- ule (CTM). CTM automatically generates en- tailed (or similar-intent) questions for a source QA pair and fine-tunes the VQA model if the VQA’s answer to the entailed question is con- sistent with the source QA pair. We demon- strate that our CTM-based training improves the consistency of VQA models on the Con- VQA datasets and is a strong baseline for fur- ther research. 1 Introduction “A skeptic, I would ask for consistency first of all.” Sylvia Plath (Plath, 2007) Visual Question Answering (VQA) (Antol et al., 2015) involves answering natural language ques- tions about images. Despite the recent progress on VQA, we observe that existing methods are prone to making blatant mistakes while answering questions regarding the same visual fact but from slightly different perspectives (Figure 1). This Figure 1: Current VQA models often fail at consistently an- swering semantically rephrased questions. To address this limitation, we construct a consistent VQA (ConVQA) dataset with diverse QA pairs that query the same visual fact. We also propose a Consistency Teacher Module (CTM) that improves VQA consistency by rewarding consistent behavior. reveals a critical limitation of the state-of-the-art models in maintaining consistency. In particular, we motivate our definition of con- sistency based on classical deductive logic (Tarski and Tarski, 1994) that defines a consistent theory as one that does not entail a contradiction. Cor- respondingly, we define consistency, in the con- text of VQA, as being able to answer questions posed from different semantic perspectives about a certain fact without any contradiction. In addi- tion, consistent Question-Answer (QA) pairs can be derived based on simple notions of logic or by commonsense reasoning. For instance, say an image contains a “large building”. Logic-based QA pairs …

Content truncated to 500 words