Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6417–6431 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6417 Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering Man Luo∗ Yankai Zeng∗ Pratyay Banerjee Chitta Baral Arizona State University {mluo26, yzeng55, pbanerj6, chitta}@asu.edu Abstract Knowledge-based visual question answering (VQA) requires answering questions with ex- ternal knowledge in addition to the content of images. One dataset that is mostly used in evaluating knowledge-based VQA is OK- VQA, but it lacks a gold standard knowledge corpus for retrieval. Existing work leverage different knowledge bases (e.g., ConceptNet and Wikipedia) to obtain external knowledge. Because of varying knowledge bases, it is hard to fairly compare models’ performance. To address this issue, we collect a natural lan- guage knowledge base that can be used for any VQA system. Moreover, we propose a Visual Retriever-Reader pipeline to approach knowledge-based VQA. The visual retriever aims to retrieve relevant knowledge, and the visual reader seeks to predict answers based on given knowledge. We introduce various ways to retrieve knowledge using text and im- ages and two reader styles: classification and extraction. Both the retriever and reader are trained with weak supervision. Our experi- mental results show that a good retriever can significantly improve the reader’s performance on the OK-VQA challenge. The code and cor- pus are provided in this link. 1 Introduction Knowledge-based VQA is a challenging task, where knowledge present in an image is not suf- ficient to answer a question. It requires a method to seek external knowledge. Figure 1 shows two examples from the OK-VQA benchmark (Marino et al., 2019), which is normally used to study knowledge-based VQA. In each of the two exam- ples, external knowledge is needed to answer the question. For instance, in the first example, to iden- tify the vehicle used in the item shown in the image (top-left), a system needs to first ground the re- ferred item as a fire hydrant and then seek external *Equal contribution Question: What sort of vehicle used this item? Answer: fire truck LXMERT: truck LXMERT + Caption: fire truck Ours: fire truck kn: fire engine, also called fire truck, mobile (nowadays self- propelled) piece of equipment used in firefighting.... Caption: a red fire hydrant sitting on the side of a road. Question: Where did this sport originate? Answer: australia, hawaii, polynesian LXMERT: california LXMERT + Caption: california Ours: hawaii kn: surfing was invented in hawaii... Caption: a man riding a wave on a surfboard in the ocean. Figure 1: Two examples from OK-VQA: the middle column are predictions by two baselines and one by our proposed Visual-Retriever-Reader pipeline. The left column is relevant knowledge and the corresponding captioning of images. knowledge presented top-right of the image. The challenge is to ground the referred object in the image and retrieve relevant knowledge where the answer is present. Although the OK-VQA benchmark encourages a VQA system to rely on external resources to answer the question, it does not provide a knowledge cor- pus for …

Content truncated to 500 words