Analyzing the Behavior of Visual Question Answering Models

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1955–1960, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Analyzing the Behavior of Visual Question Answering Models Aishwarya Agrawal∗, Dhruv Batra†,∗, Devi Parikh†,∗ ∗Virginia Tech †Georgia Institute of Technology {aish, dbatra, parikh}@vt.edu Abstract Recently, a number of deep-learning based models have been proposed for the task of Visual Question Answering (VQA). The per- formance of most models is clustered around 60-70%. In this paper we propose system- atic methods to analyze the behavior of these models as a first step towards recognizing their strengths and weaknesses, and identifying the most fruitful directions for progress. We an- alyze two models, one each from two ma- jor classes of VQA models – with-attention and without-attention and show the similari- ties and differences in the behavior of these models. We also analyze the winning entry of the VQA Challenge 2016. Our behavior analysis reveals that despite re- cent progress, today’s VQA models are “my- opic” (tend to fail on sufficiently novel in- stances), often “jump to conclusions” (con- verge on a predicted answer after ‘listening’ to just half the question), and are “stubborn” (do not change their answers across images). 1 Introduction Visual Question Answering (VQA) is a recently- introduced (Antol et al., 2015; Geman et al., 2014; Malinowski and Fritz, 2014) problem where given an image and a natural language question (e.g., “What kind of store is this?”, “How many people are waiting in the queue?”), the task is to automati- cally produce an accurate natural language answer (“bakery”, “5”). A flurry of recent deep-learning based models have been proposed for VQA (Antol et al., 2015; Chen et al., 2015; Yang et al., 2016; Xu and Saenko, 2016; Jiang et al., 2015; Andreas et al., 2016a; Wang et al., 2015; Kafle and Kanan, 2016; Lu et al., 2016; Andreas et al., 2016b; Shih et al., 2016; Kim et al., 2016; Fukui et al., 2016; Noh and Han, 2016; Ilievski et al., 2016; Wu et al., 2016; Xiong et al., 2016; Zhou et al., 2015; Saito et al., 2016). Curiously, the performance of most methods is clustered around 60-70% (com- pared to human performance of 83% on open-ended task and 91% on multiple-choice task) with a mere 5% gap between the top-9 entries on the VQA Chal- lenge 2016.1 It seems clear that as a first step to understand these models, to meaningfully compare strengths and weaknesses of different models, to de- velop insights into their failure modes, and to iden- tify the most fruitful directions for progress, it is cru- cial to develop techniques to understand the behav- ior of VQA models. In this paper, we develop novel techniques to characterize the behavior of VQA models. As con- crete instantiations, we analyze two VQA models (Lu et al., 2015; Lu et al., 2016), one each from two major classes of VQA models – with-attention and without-attention. We also analyze the winning en- try (Fukui et al., 2016) …

Content truncated to 500 words