All You May Need for VQA are Image Captions

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1947 - 1963 July 10-15, 2022 ©2022 Association for Computational Linguistics All You May Need for VQA are Image Captions Soravit Changpinyo∗ Doron Kukliansky∗ Idan Szpektor Xi Chen Nan Ding Radu Soricut Google Research {schangpi,doronk,szpektor,chillxichen,dingnan,rsoricut}@google.com Abstract Visual Question Answering (VQA) has bene- fited from increasingly sophisticated models, but has not enjoyed the same level of engage- ment in terms of data creation. In this paper, we propose a method that automatically derives VQA examples at volume, by leveraging the abundance of existing image-caption annota- tions combined with neural models for textual question generation. We show that the resulting data is of high-quality. VQA models trained on our data improve state-of-the-art zero-shot accuracy by double digits and achieve a level of robustness that lacks in the same model trained on human-annotated VQA data. 1 Introduction Visual Question Answering (VQA) is a complex multimodal task that, to be successfully modeled and evaluated, requires large amounts of annota- tions that are not naturally produced by existing business processes, the way translation-pair anno- tations (Guo et al., 2018) or image alt-text annota- tions (Sharma et al., 2018) are produced. At present, a main bottleneck for developing ro- bust VQA systems that are useful for downstream applications, such as for visually-impaired people and in the medical and education domains, appears to be a lack of large image-question-answer train- ing triplets (on the order of millions). Manual an- notation of such triplets is costly, time-consuming, and prone to a variety of human biases that are dif- ficult to account for (Yuan, 2021). In addition, the brittleness of VQA systems trained on such man- ual annotations is well-understood and documented (Agrawal et al., 2018; Kafle and Kanan, 2017). To address the data limitation, we turn to a po- tential source for creating VQA examples: image- English caption pairs (Chen et al., 2015; Sharma et al., 2018). Large-scale image caption datasets ∗ Equal contribution Generated Question Answer What are the two animals laying on the ice? “bears” What are the bears doing? “laying down” How many bears are laying on the ice? “two” How many people are sitting down? “zero” Caption: Two bears are laying down on the ice. VQ2A Figure 1: Given an English caption (along with its corre- sponding image), our VQ2A method generates high-quality question-answer pairs. These image-question-answer triplet data can be automatically produced at volume (millions of examples) and used to effectively train VQA systems. exist with millions (Changpinyo et al., 2021), sev- eral hundreds millions (Radford et al., 2021), or even billions (Jia et al., 2021) of examples. Cap- tions come mostly in the form of declarative sen- tences, e.g., “two bears are laying down on the ice”. Yet, the task of converting declarative cap- tions into VQA question/answer pairs is still largely unexplored. It requires automatically inducing can- didate answers fitting the VQA task, along with their respective questions based on the …

Content truncated to 500 words