Retrieval Augmented Visual Question Answering with Outside Knowledge

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11238 - 11254 December 7-11, 2022 ©2022 Association for Computational Linguistics Retrieval Augmented Visual Question Answering with Outside Knowledge Weizhe Lin Department of Engineering University of Cambridge United Kingdom wl356@cam.ac.uk Bill Byrne Department of Engineering University of Cambridge United Kingdom bill.byrne@eng.cam.ac.uk Abstract Outside-Knowledge Visual Question Answer- ing (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to an- swer questions about images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve documents from external knowledge bases, such as Wikipedia, but with DPR trained separately from answer generation, introducing a potential limit on the overall system perfor- mance. Instead, we propose a joint training scheme which includes differentiable DPR in- tegrated with answer generation so that the sys- tem can be trained in an end-to-end fashion. Our experiments show that our scheme out- performs recent OK-VQA systems with strong DPR for retrieval. We also introduce new di- agnostic metrics to analyze how retrieval and generation interact. The strong retrieval ability of our model significantly reduces the num- ber of retrieved documents needed in training, yielding significant benefits in answer quality and computation required for training. 1 Introduction Visual Question Answering (VQA) is a challenging problem that lies at the intersection of Computer Vision, Natural Language Processing, and Infor- mation Retrieval. The objective in VQA is to read an image and provide an answer to an accompa- nying question about the image content. Current approaches to VQA employ deep-learning-based systems to jointly understand images and text. VQA is particularly challenging when the an- swer to the question is not directly available in the image. In Knowledge-based VQA (KB-VQA), the VQA system must access external knowledge sources to find a correct and complete answer. The Ouside-Knowledge VQA task (OK-VQA) (Marino et al., 2019) consists of questions that requires gen- eral knowledge and simple inference to answer (Fig. 1). Such questions are even hard for humans. Unlike other KB-VQA datasets (e.g. FVQA (Wang et al., 2017)) which provide an associated knowl- edge base, OK-VQA encourages using any outside knowledge in answering questions. Question : How many teeth does this animal use to have? Answer: 26 Figure 1: OK-VQA contains questions whose answer cannot be found within the image. The need to adapt and refresh knowledge sources motivates the study of KB-VQA systems that can extract knowledge from both structured (e.g. Con- ceptNet (Speer et al., 2017)) and unstructured knowledge representations (e.g. Wikipedia pas- sages). Recent designs (Luo et al., 2021; Gao et al., 2022) approach VQA in two distinct steps: (1) Knowledge Retrieval extracts documents from a large knowledge base; (2) Answer Generation pro- duces an answer from these documents. Knowl- edge Retrieval can be done via Dense Passage Re- trieval (DPR) (Karpukhin et al., 2020), which con- sists of a question encoder and a document encoder (both Transformer-based) that encode questions and documents into separate dense representations. The DPR system is trained to assign higher scores …

Content truncated to 500 words