Augmenting Image Question Answering Dataset by Exploiting Image Captions

Augmenting Image Question Answering Dataset by Exploiting Image Captions Masashi Yokota, Hideki Nakayama Graduate School of Information Science and Technology, The University of Tokyo Tokyo, JP {yokota, nakayama}@nlab.ci.i.u-tokyo.ac.jp Abstract Image question answering (IQA) is one of the tasks that need rich resources, i.e. supervised data, to achieve optimal performance. However, because IQA is a challenging task that handles complex input and output information, the cost of naive manual annotation can be prohibitively expensive. On the other hand, it is thought to be relatively easy to obtain relevant pairs of an image and text in an unsupervised manner (e.g., crawling Web data). Based on this expectation, we propose a framework to augment training data for IQA by generating additional examples from unannotated pairs of an image and captions. The important constraint that a generated IQA example must satisfy is that its answer must be inferable from the corresponding image and question. To satisfy this, we first select a possible answer for a given image by randomly extracting an answer from corresponding captions. Then we generate the question from the triplets of the image, captions and fixed answer. In experiments, we test our method on the Visual Genome dataset varying the ratio of seed supervised data and demonstrate its effectiveness. Keywords: Semi-Supervised Learning, Self-labeling, Image Question Generation, Image Question Answering 1. Introduction The objective of image question answering (IQA) is to pre- dict a correct answer given an image and a question. To achieve a satisfactory performance, we generally require a large amount of human-annotated data which is expensive to prepare. One of the possible ways to mitigate the bur- den of collecting annotations is generating a pseudo dataset, which is one of the effective approaches in semi-supervised learning. Although this approach has been well-known to benefit a variety of tasks such as image classification (Lee, 2013), neural machine translation (He et al., 2016), and reading comprehension (Yang et al., 2017), its effec- tiveness has not yet been demonstrated for IQA. In this study, we address the resource problem in IQA by proposing a framework of pseudo data generation exploit- ing captions as auxiliary information. In IQA, a generated pseudo dataset that comprises triplets consisting of an im- age I, question Q, and answer A should satisfy an impor- tant constraint: the components of a triplet (I, Q, A) must be relevant to each other. In other words, the answer A must be inferable from an image-question pair (I, Q). We ad- dress the problem of component relevancy by focusing on text captions C as the additional information. Our expecta- tion is that pairs of mutually relevant image and captions are relatively easily obtainable from the Web (e.g. Wikipedia, BBC News, etc.). These captions are expected to contain information in images such as object colors, human actions, relationships between objects, etc. We first fix the answer A by sampling a token from captions as a possible answer for an image, and then generate a question conditioned by the triplet (I, C, A). …

Content truncated to 500 words