MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5558 - 5570 December 7-11, 2022 ©2022 Association for Computational Linguistics MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, William W. Cohen Google Research {wenhuchen,hexiang,patverga,wcohen}@google.com Abstract While language Models store a massive amount of world knowledge implicitly in their parameters, even very large models often fail to encode information about rare entities and events, while incurring huge computational costs. Recently, retrieval-augmented models, such as REALM, RAG, and RETRO, have incorporated world knowledge into language generation by leveraging an external non- parametric index and have demonstrated im- pressive performance with constrained model sizes. However, these methods are restricted to retrieving only textual knowledge, neglect- ing the ubiquitous amount of knowledge in other modalities like images – much of which contains information not covered by any text. To address this limitation, we propose the first Multimodal Retrieval-Augmented Trans- former (MuRAG), which accesses an external non-parametric multimodal memory to aug- ment language generation. MuRAG is pre- trained with a mixture of large-scale image- text and text-only corpora using a joint con- trastive and generative loss. We perform ex- periments on two different datasets that re- quire retrieving and reasoning over both im- ages and text to answer a given query: We- bQA, and MultimodalQA. Our results show that MuRAG achieves state-of-the-art accu- racy, outperforming existing models by 10- 20% absolute on both datasets and under both distractor and full-wiki settings. 1 Introduction Pre-trained language models like GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), etc have been shown to capture a massive amount of world knowledge implicitly in their parame- ters. However, using such large models incurs an extremely high computation cost. As an alterna- tive to a singular monolithic transformer, retrieval- augmented architectures like KNN-LM (Khandel- wal et al., 2019), REALM (Guu et al., 2020), Figure 1: Visual information-seeking queries: These queries are unanswerable with text-only retrieval and require retrieving and reasoning over images. RAG (Lewis et al., 2020), FiD (Izacard and Grave, 2021), and RETRO (Borgeaud et al., 2021) have been proposed to decouple world knowledge from the model’s parameters. More specifically, these models are trained to access an external mem- ory to enhance the model’s predictions. Such retrieval-augmented architectures have multiple beneficial properties including: decreased model size (Borgeaud et al., 2021), better attribution/- explanation for model predictions (Lewis et al., 2020), and adaptability to new information with- out retraining (Verga et al., 2021). However, pre- vious retrieval-augmented models are limited to memories that contain only text or structured data and hence cannot make use of the massive amount of multimodal knowledge available on the web— much of which contains information only available in non-text modalities. Figure 1, shows several information-seeking queries that require retrieving and reasoning over visual knowledge. Here, a user first poses a ques- tion such as “What can be found on the White House balconies at Christmas”. The system then …

Content truncated to 500 words