Improving Visual Question Answering by Referring to Generated Paragraph Captions

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3606–3612 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3606 Improving Visual Question Answering by Referring to Generated Paragraph Captions Hyounghun Kim UNC Chapel HillMohit Bansal {hyounghk, mbansal}@cs.unc.edu Abstract Paragraph-style image captions describe di- verse aspects of an image as opposed to the more common single-sentence captions that only provide an abstract description of the image. These paragraph captions can hence contain substantial information of the image for tasks such as visual question answering. Moreover, this textual information is comple- mentary with visual information present in the image because it can discuss both more ab- stract concepts and more explicit, intermediate symbolic information about objects, events, and scenes that can directly be matched with the textual question and copied into the tex- tual answer (i.e., via easier modality match). Hence, we propose a combined Visual and Textual Question Answering (VTQA) model which takes as input a paragraph caption as well as the corresponding image, and answers the given question based on both inputs. In our model, the inputs are fused to extract re- lated information by cross-attention (early fu- sion), then fused again in the form of consen- sus (late fusion), and finally expected answers are given an extra score to enhance the chance of selection (later fusion). Empirical results show that paragraph captions, even when auto- matically generated (via an RL-based encoder- decoder model), help correctly answer more visual questions. Overall, our joint model, when trained on the Visual Genome dataset, significantly improves the VQA performance over a strong baseline model. 1 Introduction Understanding visual information along with nat- ural language have been studied in different ways. In visual question answering (VQA) (Antol et al., 2015; Goyal et al., 2017; Lu et al., 2016; Fukui et al., 2016; Xu and Saenko, 2016; Yang et al., 2016; Zhu et al., 2016; Anderson et al., 2018), models are trained to choose the correct answer given a question about an image. On the other hand, in image captioning tasks (Karpathy and Fei-Fei, 2015; Johnson et al., 2016; Anderson et al., 2018; Krause et al., 2017; Liang et al., 2017; Melas-Kyriazi et al., 2018), the goal is to generate sentences which should describe a given image. Similar to the VQA task, image captioning mod- els should also learn the relationship between par- tial areas in an image and the generated words or phrases. While these two tasks seem to have dif- ferent directions, they have the same purpose: un- derstanding visual information with language. If their goal is similar, can the tasks help each other? In this work, we propose an approach to im- prove a VQA model by exploiting textual infor- mation from a paragraph captioning model. Sup- pose you are assembling furniture by looking at a visual manual. If you are stuck at a certain step and you are given a textual manual which more ex- plicitly describes the names and shapes of …

Content truncated to 500 words