Generating Question Relevant Captions to Aid Visual Question Answering

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3585–3594 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3585 Generating Question Relevant Captions to Aid Visual Question Answering Jialin Wu, Zeyuan Hu and Raymond J. Mooney Department of Computer Science University of Texas at Austin {jialinwu, iamzeyuanhu, mooney}@cs.utexas.edu Abstract Visual question answering (VQA) and im- age captioning require a shared body of gen- eral knowledge connecting language and vi- sion. We present a novel approach to improve VQA performance that exploits this connec- tion by jointly generating captions that are tar- geted to help answer a specific visual ques- tion. The model is trained using an exist- ing caption dataset by automatically determin- ing question-relevant captions using an on- line gradient-based method. Experimental re- sults on the VQA v2 challenge demonstrates that our approach obtains state-of-the-art VQA performance (e.g. 68.4% on the Test-standard set using a single model) by simultaneously generating question-relevant captions. 1 Introduction In recent years, visual question answering (VQA) (Antol et al., 2015) and image captioning (Don- ahue et al., 2015; Rennie et al., 2017) have been widely studied in both the computer vision and NLP communities. Most recent VQA research (Lu et al., 2017; Pedersoli et al., 2017; Anderson et al., 2018; Lu et al., 2018) concentrates on directly uti- lizing visual input features including detected ob- jects, attributes, and relations between pairs of ob- jects. However,little VQA research works on exploit- ing textual features from the image which are able to tersely encode the necessary information to an- swer the questions. This information could be richer than the visual features in that the sentences have fewer structural constraints and can easily in- clude the attributes of and relation among multi- ple objects. In fact, we observe that appropriate captions can be very useful for many VQA ques- tions. In particular, we trained a model to answer visual questions for the VQA v2 challenge (An- tol et al., 2015) only using the human annotated Human Captions : 1) A man on a blue surfboard on top of some rough water. 2) A young surfer in a wetsuit surfs a small wave. 3) A young man rides a surf board on a small wave while a man swims in the background. 4) A young man is on his surf board with someone in the background. 5) A boy riding waves on his surf board in the ocean. Question 1: Does this boy have a full wetsuit on? Caption: A young man wearing wetsuit surfing on a wave. Question 2: What color is the board? Caption: A young man riding a wave on a blue surfboard. Figure 1: Examples of our generated question-relevant captions. During the training phase, our model selects the most relevant human captions for each question (marked by the same color). captions without images and achieved a score of 59.6%, outperforming a large number of VQA models that use image features. Existing work using captions for …

Content truncated to 500 words