Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 919–924, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions Arijit Ray1, Gordon Christie1, Mohit Bansal2, Dhruv Batra3,1, Devi Parikh3,1 1Virginia Tech 2UNC Chapel Hill 3Georgia Institute of Technology {ray93,gordonac,dbatra,parikh}@vt.edu mbansal@cs.unc.edu Abstract Visual Question Answering (VQA) is the task of answering natural-language questions about images. We introduce the novel prob- lem of determining the relevance of questions to images in VQA. Current VQA models do not reason about whether a question is even related to the given image (e.g., What is the capital of Argentina?) or if it requires infor- mation from external resources to answer cor- rectly. This can break the continuity of a dia- logue in human-machine interaction. Our ap- proaches for determining relevance are com- posed of two stages. Given an image and a question, (1) we first determine whether the question is visual or not, (2) if visual, we de- termine whether the question is relevant to the given image or not. Our approaches, based on LSTM-RNNs, VQA model uncertainty, and caption-question similarity, are able to outper- form strong baselines on both relevance tasks. We also present human studies showing that VQA models augmented with such question relevance reasoning are perceived as more in- telligent, reasonable, and human-like. 1 Introduction Visual Question Answering (VQA) is the task of predicting a suitable answer given an image and a question about it. VQA models (e.g., (Antol et al., 2015; Ren et al., 2015)) are typically discriminative models that take in image and question representa- tions and output one of a set of possible answers. Our work is motivated by the following key ob- servation – all current VQA systems always output an answer regardless of whether the input question makes any sense for the given image or not. Fig. 1 Non-Visual Visual True-Premise Who is the president of the USA? What is the girl wearing? What is the cat wearing? Visual False-Premise Figure 1: Example irrelevant (non-visual, false-premise) and relevant (visual true-premise) questions in VQA. shows examples of relevant and irrelevant questions. When VQA systems are fed irrelevant questions as input, they understandably produce nonsensical an- swers (Q: “What is the capital of Argentina?” A: “fire hydrant”). Humans, on the other hand, are unlikely to provide such nonsensical answers and will instead answer that this is irrelevant or use an- other knowledge source to answer correctly, when possible. We argue that this implicit assumption by all VQA systems – that an input question is always relevant for the input image – is simply untenable as VQA systems move beyond standard academic datasets to interacting with real users, who may be unfamiliar, or malicious. The goal of this work is to make VQA systems more human-like by providing them the capability to identify relevant questions. While existing work has reasoned about cross- modal similarity, being able to identify whether a question is relevant to a given …

Content truncated to 500 words