Multi-grained Attention with Object-level Grounding for Visual Question Answering

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3595–3600 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3595 Multi-grained Attention with Object-level Grounding for Visual Question Answering Pingping Huang, Jianhui Huang, Yuqing Guo, Min Qiao and Yong Zhu Baidu Inc., Beijing, China {huangpingping,huangjianhui,guoyuqing,qiaomin,zhuyong}@baidu.com Abstract Attention mechanisms are widely used in Vi- sual Question Answering (VQA) to search for visual clues related to the question. Most ap- proaches train attention models from a coarse- grained association between sentences and im- ages, which tends to fail on small objects or uncommon concepts. To address this problem, this paper proposes a multi-grained attention method. It learns explicit word- object correspondence by two types of word- level attention complementary to the sentence- image association. Evaluated on the VQA benchmark, the multi-grained attention model achieves competitive performance with state- of-the-art models. And the visualized atten- tion maps demonstrate that addition of object- level groundings leads to a better understand- ing of the images and locates the attended ob- jects more precisely. 1 Introduction Visual Question Answering (Antol et al., 2015; Goyal et al., 2017a) is a multi-modal task requir- ing to provide an answer to the question with ref- erence to a given image. Most current VQA sys- tems resort to deep neural networks and solve the problem by end-to-end learning. First the question and the image are encoded into semantic represen- tations independently. Then the multi-modal fea- tures are fused into one unified representation for which the answer is predicted (Malinowski et al., 2015; Fukui et al., 2016; Anderson et al., 2018). A key point to a successful VQA system is to discover the most relevant image regions to the question. This is commonly resolved by attention mechanisms, where a spatial attention distribution highlighting the visual focus is computed accord- ing to the similarity between the whole question and image regions (Xu et al., 2015; Yang et al., 2016; Lu et al., 2016). Although such coarse Q: What is the man wearing around his face? A: glasses Up-down Model: nothing Our Model: glasses Figure 1: An example of VQA and the attention maps produced by a state-of-the-art model and our model. sentence-image alignment reports promising re- sults in general, it sometimes fails to locate small objects or understand a complicated scenario. For the example in Figure 1, the question is “What is the man wearing around his face”. Human has no difficulty in finding the visual clue on the people’s faces, and accordingly provide the correct answer “glasses”. However, by visualizing the attention map of a state-of-the-art VQA model, we find that the attention is mistakenly focused on the men’s body rather than their faces. In order to identify related objects more pre- cisely, this paper proposes a multi-grained atten- tion mechanism that involves object-level ground- ing complementary to the sentence-image associ- ation. Specifically, a matching model is trained on an object-detection dataset to learn explicit corre- spondence between the content words in the …

Content truncated to 500 words