End-to-end Knowledge Retrieval with Multi-modal Queries

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8573–8589 July 9-14, 2023 ©2023 Association for Computational Linguistics End-to-end Knowledge Retrieval with Multi-modal Queries Man Luo1 Zhiyuan Fang2 Tejas Gokhale1 Yezhou Yang1 Chitta Baral1 1 Arizona State University 2 Amazon Alexa {mluo26, tgokhale, yz.yang, chitta}@asu.edu, zyfang@amazon.com Abstract We investigate knowledge retrieval with multi- modal queries, i.e. queries containing informa- tion split across image and text inputs, a chal- lenging task that differs from previous work on cross-modal retrieval. We curate a new dataset called ReMuQ 1 for benchmarking progress on this task. ReMuQ requires a system to retrieve knowledge from a large corpus by integrating contents from both text and image queries. We introduce a retriever model “ReViz” that can di- rectly process input text and images to retrieve relevant knowledge in an end-to-end fashion without being dependent on intermediate mod- ules such as object detectors or caption gen- erators. We introduce a new pretraining task that is effective for learning knowledge retrieval with multimodal queries and also improves per- formance on downstream tasks. We demon- strate superior performance in retrieval on two datasets (ReMuQ and OK-VQA) under zero- shot settings as well as further improvements when finetuned on these datasets. 1 Introduction Humans recall, retrieve, and communicate infor- mation using many indirect hints and cues. For instance, if we want to explain the concept of a “leopard” but have forgotten the name, we can re- late the concept to a picture of a tiger and say “it is an animal that looks like this, but has spots in- stead of stripes”. Similarly, when children learn to draw a new shape like an oval, teachers often prompt them by showing a circle, but saying “make the circle stretched-out”. This method of learning new concepts from visual aids and language de- scriptions is a common way of reinforcing existing knowledge and allowing learners to explore and retrieve new concepts (Kinder, 1942). We propose a task for vision-language models to retrieve knowledge with multi-modal queries, 1pronounced re–µ-queue. Data and code: https:// github.com/luomancs/ReMuQ. i.e. queries in which hints about the information to be retrieved are split across image and text inputs. Figure 1 contains an example of this task, where the image shows the Empire State Building in New York City. If we retrieve knowledge using only the image, is it likely that the retrieved information (K1) will be related to the Empire State Building. However, K1 is insufficient to answer the question. On the other hand, if we retrieve knowledge using only the question, then the information retrieved (K2) is likely to be related to the tallest building in all cities (and not restricted to New York City). K2 by itself is also insufficient to answer the question. This example shows that the combined query con- taining both image and text (question) is necessary for retrieving relevant knowledge (K3). We introduce a new benchmark and dataset called ReMuQ (Retrieval with Multimodal Queries) to train …

Content truncated to 500 words