Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval

Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval Gregor Geigle∗1, Jonas Pfeiffer∗1, Nils Reimers1, Ivan Vuli´c2, Iryna Gurevych1 1Ubiquitous Knowledge Processing Lab, Technical University of Darmstadt, Germany 2Language Technology Lab, University of Cambridge, United Kingdom www.ukp.tu-darmstadt.de Abstract Current state-of-the-art approaches to cross- modal retrieval process text and visual input jointly, relying on Transformer-based archi- tectures with cross-attention mechanisms that attend over all words and objects in an image. While offering unmatched retrieval perfor- mance, such models: 1) are typically pretrained from scratch and thus less scalable, 2) suffer from huge retrieval latency and inefficiency issues, which makes them impractical in realistic applications. To address these crucial gaps towards both improved and efficient cross- modal retrieval, we propose a novel fine-tuning framework that turns any pretrained text-image multi-modal model into an efficient retrieval model. The framework is based on a cooper- ative retrieve-and-rerank approach that com- bines: 1) twin networks (i.e., a bi-encoder) to separately encode all items of a corpus, enabling efficient initial retrieval, and 2) a cross-encoder component for a more nuanced (i.e., smarter) ranking of the retrieved small set of items. We also propose to jointly fine- tune the two components with shared weights, yielding a more parameter-efficient model. Our experiments on a series of standard cross-modal retrieval benchmarks in mono- lingual, multilingual, and zero-shot setups, demonstrate improved accuracy and huge effi- ciency benefits over the state-of-the-art cross- encoders.1 1 Introduction Information-rich and efficient methods for deal- ing with large unstructured data in both computer vision and NLP are required to process and under- stand huge amounts of user-created content and beyond. In multi-modal contexts, such methods enable fundamental applications such as image ∗Both authors contributed equally to this work. 1We release the code and model weights at github .com/UKPLab/MMT-Retrieval. retrieval. A typical efficient bi-encoder2 approach encodes images and text separately and then induces a shared high-dimensional multi-modal feature space. This enables cross-modal retrieval, where standard distance metrics identify the most similar examples for each query in the data col- lection via nearest-neighbor search (Arya et al., 1998; Kushilevitz et al., 2000; Liu et al., 2004; Andoni and Indyk, 2008; Hajebi et al., 2011). These bi-encoder approaches have already been shown to achieve reasonable performance in search and retrieval applications, both mono- lingually for English (Nam et al., 2017; Faghri et al., 2018; Zheng et al., 2020; Wang et al., 2019a; Shi et al., 2019) and in multilingual contexts (Gella et al., 2017; K´ad´ar et al., 2018; Kim et al., 2020; Wehrmann et al., 2019; Burns et al., 2020). However, they cannot match performance of more recent attention-based methods. Here, a typ- ical modus operandi is to apply a cross-attention mechanism between examples from the two modalities to compute their similarity score, re- lying on Transformer-based neural architectures (Vaswani et al., 2017). Such so-called multi-modal cross-encoders (CE) (Tan and Bansal, 2019; Lu et al., 2019; Chen et al., 2020; Li et al., 2020a; Gan et al., 2020; Li et al., 2020b; Ni et al., …

Content truncated to 500 words