Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8061 - 8072 December 7-11, 2022 ©2022 Association for Computational Linguistics Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering Jialin Wu Department of Computer Science The University of Texas at Austin jialinwu@utexas.edu Raymond J. Mooney Department of Computer Science The University of Texas at Austin mooney@cs.utexas.edu Abstract Most Outside-Knowledge Visual Question An- swering (OK-VQA) systems employ a two- stage framework that first retrieves external knowledge given the visual question and then predicts the answer based on the retrieved con- tent. However, the retrieved knowledge is often inadequate. Retrievals are frequently too gen- eral and fail to cover specific knowledge needed to answer the question. Also, the naturally available supervision (whether the passage con- tains the correct answer) is weak and does not guarantee question relevancy. To address these issues, we propose an Entity-Focused Retrieval (EnFoRe) model that provides stronger super- vision during training and recognizes question- relevant entities to help retrieve more specific knowledge. Experiments show that our En- FoRe model achieves superior retrieval per- formance on OK-VQA, the currently largest outside-knowledge VQA dataset. We also com- bine the retrieved knowledge with state-of-the- art VQA models, and achieve a new state-of- the-art performance on OK-VQA. 1 Introduction Passage retrieval under a multi-modal setting is a critical prerequisite for applications such as outside- knowledge visual question answering (OK-VQA) (Marino et al., 2019), which requires effectively uti- lizing knowledge external to the image. Recently, dense passage retrievers with deep semantic rep- resentations powered by large transformer models have shown superior performance to traditional sparse retrievers such as BM25 (Robertson and Zaragoza, 2009) and TF-IDF under both textual (Karpukhin et al., 2020; Chen et al., 2021; Lewis et al., 2022) and multi-modal settings (Luo et al., 2021; Qu et al., 2021; Gui et al., 2021). In this work, we investigate two main drawbacks of recent dense retrievers (Karpukhin et al., 2020; Chen et al., 2021; Lewis et al., 2022; Luo et al., 2021; Qu et al., 2021; Gui et al., 2021), which are Being omnivores they enjoy eating live crickets and other insects and small amounts of chopped fruits and vegetables such as … Bell pepper The bell pepper is … in different colours, including red, yellow, orange, …they are commonly used as a vegetable ingredient … Q: What holiday is this? A: Thanksgiving. Q: This plush toy was named after what US president? A: Theodore Teddy Roosevelt. critical entity: teddy bear critical entity: turkey Q: Is the large yellow object a fruit or a vegetable? A: vegetable. critical entity: bell pepper Figure 1: Top: Examples of critical entities upon which retrieval models should focus; Bottom: Example of improved passage retrieval using critical entities. typically trained to produce similar representations for input queries and passages containing ground- truth answers. First, as most retrieval models encode the query and passages as a whole, they fail to explicitly discover entities critical to answering the ques- tion (Chen et al., 2021). This …

Content truncated to 500 words