Human Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions?

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 932–937, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions? Abhishek Das1∗ Harsh Agrawal1∗ C. Lawrence Zitnick2 Devi Parikh1,3 Dhruv Batra1,3 1Virginia Tech 2Facebook AI Research 3Georgia Institute of Technology {abhshkdz, harsh92, parikh, dbatra}@vt.edu, zitnick@fb.com Abstract We conduct large-scale studies on ‘human at- tention’ in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We de- sign and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred im- age to answer a question. Thus, we in- troduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against hu- man attention both qualitatively (via visualiza- tions) and quantitatively (via rank-order cor- relation). Overall, our experiments show that current VQA attention models do not seem to be looking at the same regions as humans. 1 Introduction It helps to pay attention. Humans have the ability to quickly perceive a scene by selectively attending to parts of the image instead of processing the whole scene in its entirety (Rensink, 2000). Inspired by hu- man attention, a recent trend in computer vision and deep learning is to build computational models of at- tention. Given an input signal, these models learn to attend to parts of it for further processing and have been successfully applied in machine transla- tion (Bahdanau et al., 2015; Firat et al., 2016), ob- ject recognition (Ba et al., 2015; Mnih et al., 2014; Sermanet et al., 2014), image captioning (Xu et al., 2015; Cho et al., 2015) and visual question answer- ing (Yang et al., 2016; Lu et al., 2016; Xu and Saenko, 2015; Xiong et al., 2016). In this work, we study attention for the task of Vi- sual Question Answering (VQA). Unlike image cap- tioning, where a coarse understanding of an image ∗Denotes equal contribution. Figure 1: Different human attention regions based on question. (best viewed in color) is often sufficient for producing generic descriptions (Devlin et al., 2015), visual questions selectively tar- get different areas of an image including background details and underlying context. This suggests that a VQA model may benefit from an explicit or implicit attention mechanism to answer a question correctly. In this work, we are interested in the following ques- tions: 1) Which image regions do humans choose to look at in order to answer questions about images? 2) Do deep VQA models with attention mechanisms attend to the same regions as humans? We design and conduct studies to collect “human attention maps”. Figure 1 shows human attention maps on the same image for two different ques- tions. When asked ‘What type is the surface?’, hu- mans choose to look at the floor, while attention for ‘Which game is being played?’ is concentrated around the player and racket. These human attention maps can be …

Content truncated to 500 words