Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6346–6361 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6346 Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering Jihyung Kil, Cheng Zhang, Dong Xuan, Wei-Lun Chao The Ohio State University, Columbus, OH, USA {kil.5, zhang.7804, xuan.3, chao.209}@osu.edu Abstract Visual question answering (VQA) is challeng- ing not only because the model has to handle multi-modal information, but also because it is just so hard to collect sufficient training exam- ples — there are too many questions one can ask about an image. As a result, a VQA model trained solely on human-annotated examples could easily over-fit specific question styles or image contents that are being asked, leaving the model largely ignorant about the sheer di- versity of questions. Existing methods address this issue primarily by introducing an auxiliary task such as visual grounding, cycle consis- tency, or debiasing. In this paper, we take a drastically different approach. We found that many of the “unknowns” to the learned VQA model are indeed “known” in the dataset im- plicitly. For instance, questions asking about the same object in different images are likely paraphrases; the number of detected or anno- tated objects in an image already provides the answer to the “how many” question, even if the question has not been annotated for that image. Building upon these insights, we present a sim- ple data augmentation pipeline SIMPLEAUG to turn this “known” knowledge into train- ing examples for VQA. We show that these augmented examples can notably improve the learned VQA models’ performance, not only on the VQA-CP dataset with language prior shifts but also on the VQA v2 dataset with- out such shifts. Our method further opens up the door to leverage weakly-labeled or unla- beled images in a principled way to enhance VQA models. Our code and data are pub- licly available at https://github.com/ heendung/simpleAUG. 1 Introduction “A picture is worth a thousand words,” which tells how expressive an image can be, but also how chal- lenging it is to teach a machine to understand an im- age like we humans do. Visual question answering (VQA) (Antol et al., 2015; Goyal et al., 2017; Zhu Q: Where are the napkins? Q: What is the oven made of? Q: Is the dispenser beneath the microware full? A: Table A: Stainless steel A: No Original QA Pairs Q: How many bottles are visible? Q: Is this a modern microwave? Q: What color is the floor? A: 1 A: Yes A: Brown Q: What is the oven made from? Propagation QA Pairs Paraphrasing QA Pairs Q: What is the oven made from? A: Stainless steel SIMPLEAUG Figure 1: Illustration of our approach SIMPLEAUG. We show a training image and its corresponding question-answer pairs in VQA v2 (Goyal et al., 2017), and our generated pairs. A VQA model (Anderson et al., 2018) trained on the original dataset just cannot answer these new questions …

Content truncated to 500 words