WeaQA: Weak Supervision via Captions for Visual Question Answering

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3420–3435 August 1–6, 2021. ©2021 Association for Computational Linguistics 3420 WeaQA: Weak Supervision via Captions for Visual Question Answering Pratyay Banerjee Tejas Gokhale Yezhou Yang Chitta Baral Arizona State University pbanerj6, tgokhale, yz.yang, chitta@asu.edu Abstract Methodologies for training visual question an- swering (VQA) models assume the availabil- ity of datasets with human-annotated Image- Question-Answer (I-Q-A) triplets. This has led to heavy reliance on datasets and a lack of generalization to new types of questions and scenes. Linguistic priors along with biases and errors due to annotator subjectivity have been shown to percolate into VQA models trained on such samples. We study whether models can be trained without any human-annotated Q-A pairs, but only with images and their as- sociated textual descriptions or captions. We present a method to train models with syn- thetic Q-A pairs generated procedurally from captions. Additionally, we demonstrate the ef- ficacy of spatial-pyramid image patches as a simple but effective alternative to dense and costly object bounding box annotations used in existing VQA models. Our experiments on three VQA benchmarks demonstrate the effi- cacy of this weakly-supervised approach, espe- cially on the VQA-CP challenge, which tests performance under changing linguistic priors. 1 Introduction Since Visual Question Answering (VQA) was first proposed as a Turing test (Malinowski and Fritz, 2014), several human-annotated datasets (Mo- gadala et al., 2019) have been used to train and evaluate VQA models. Unfortunately, heavy re- liance on these datasets for training has the un- wanted side-effects of bias towards answer styles, question-types (Chao et al., 2018), and spurious correlations with language priors (Agrawal et al., 2018). Similar findings have been reported for nat- ural language tasks (Gururangan et al., 2018; Niven and Kao, 2019; Kaushik et al., 2020). Evaluating VQA models on test-sets that are very similar to training sets is deceptive and inadequate and not an accurate measure of robustness. To address this, one line of work has focused on balancing, de-biasing, and diversifying sam- ples (Goyal et al., 2017; Zhang et al., 2016). How- ever, crowd-sourcing “unbiased” labels is difficult and costly; it requires a well-designed annotation interface and a large-scale annotation effort with dedicated and able annotators (Sakaguchi et al., 2020). The alternative (that this paper aligns itself with) is to avoid the use of explicit human anno- tations and instead to train models in an unsuper- vised manner by synthesizing training data. These techniques, coined unsupervised1, come with many advantages – human bias and subjectivity are re- duced; the techniques are largely domain-agnostic and can be transferred from one language to an- other (low resource languages) or from one visual domain to another. For instance, template-based Q- A generation developed for synthetic blocks-world images in CLEVR (Johnson et al., 2017) can also be used to generate Q-A pairs for natural complex scenes in GQA (Hudson and Manning, 2019) or the referring-expressions task (Liu et al., 2019). In this work, we train VQA models without using human-annotated Q-A pairs. …

Content truncated to 500 words