The Promise of Premise: Harnessing Question Premises in Visual Question Answering

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 926–935 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics The Promise of Premise: Harnessing Question Premises in Visual Question Answering Aroma Mahendru∗,1 Viraj Prabhu∗,1 Akrit Mohapatra∗,1 Dhruv Batra2 Stefan Lee1 1Virginia Tech 2Georgia Institute of Technology {maroma, virajp, akrit}@vt.edu, dbatra@gatech.edu, steflee@vt.edu Abstract In this paper, we make a simple observa- tion that questions about images often con- tain premises – objects and relationships implied by the question – and that reason- ing about premises can help Visual Ques- tion Answering (VQA) models respond more intelligently to irrelevant or previ- ously unseen questions. When presented with a question that is ir- relevant to an image, state-of-the-art VQA models will still answer purely based on learned language biases, resulting in non- sensical or even misleading answers. We note that a visual question is irrelevant to an image if at least one of its premises is false (i.e. not depicted in the image). We leverage this observation to construct a dataset for Question Relevance Predic- tion and Explanation (QRPE) by searching for false premises. We train novel question relevance detection models and show that models that reason about premises consis- tently outperform models that do not. We also find that forcing standard VQA models to reason about premises during training can lead to improvements on tasks requiring compositional reasoning. 1 Introduction The task of providing natural language answers to free-form questions about an image – i.e. Visual Question Answering (VQA) – has received sub- stantial attention in the past few years (Malinowski and Fritz, 2014; Antol et al., 2015; Malinowski et al., 2015; Zitnick et al., 2016; Kim et al., 2016; Wu et al., 2016; Lu et al., 2016; Andreas et al., ∗Denotes equal contribution. Figure 1: Questions asked about images often contain ‘premises’ that imply visual semantics. From the above ques- tion, we can infer that a relevant image must contain a man, a racket, and that the man must be holding the racket. We extract these premises from visually grounded questions and use them to construct a new dataset and models for question relevance prediction. We also find that augmenting standard VQA training with simple premise-based questions results in improvements on tasks requiring compositional reasoning. 2016; Lu et al., 2017) and has quickly become a popular problem area. Despite significant progress on VQA benchmarks (Antol et al., 2015), current models still present a number of unintelligent and problematic tendencies. When faced with questions that are irrelevant or not applicable for an image, current ‘forced choice’ models will still produce an answer. For example, given an image of a dog and a query “What color is the bird?”, standard VQA models might answer “Red” confidently, based solely on language biases in the training set (i.e. an over- abundance of the word “red”). In these cases, the predicted answers are senseless at best and mis- leading at worst, with either case posing serious problems for real-world applications. Like …

Content truncated to 500 words