Inflate and Shrink:Enriching and Reducing Interactions for Fast Text-Image Retrieval

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9796–9809 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9796 Inflate and Shrink: Enriching and Reducing Interactions for Fast Text-Image Retrieval Haoliang Liu, Tan Yu, Ping Li Cogntive Computing Lab Baidu Research No.10 Xibeiwang East Road, Beijing 100193, China 10900 NE 8th St. Bellevue, Washington 98004, USA {haoliangliu.leo,tanyuuynat,pingli98}@gmail.com Abstract By exploiting the cross-modal attention, cross- BERT methods have achieved state-of-the-art accuracy in cross-modal retrieval. Neverthe- less, the heavy text-image interactions in the cross-BERT model are prohibitively slow for large-scale retrieval. Late-interaction meth- ods trade off retrieval accuracy and efficiency by exploiting cross-modal interaction only in the late stage, attaining a satisfactory retrieval speed. In this work, we propose an inflat- ing and shrinking approach to further boost the efficiency and accuracy of late-interaction methods. The inflating operation plugs several codes in the input of the encoder to exploit the text-image interactions more thoroughly for higher retrieval accuracy. Then the shrink- ing operation gradually reduces the text-image interactions through knowledge distilling for higher efficiency. Through an inflating oper- ation followed by a shrinking operation, both efficiency and accuracy of a late-interaction model are boosted. Systematic experiments on public benchmarks demonstrate the effective- ness of our inflating and shrinking approach. 1 Introduction Efficiency and accuracy are two key factors of a re- trieval system. In many cases, designing a retrieval system is striving to balance efficiency and accu- racy. Embedding-based methods (Ordonez et al., 2011; Gong et al., 2014; Faghri et al., 2018) are early works for tackling the cross-modal retrieval. They encode each image or text into a global em- bedding. Then the text-image similarity is mea- sured by the distance between their embeddings in the learned feature space. Since there are no inter- actions between text and image, embedding-based methods only need O(N +M) computational com- plexity to encode N images and M texts. The lin- ear computational complexity of embedding-based methods makes them scalable to large-scale cross- modal retrieval. They hence have been widely de- ployed in real-world cross-modal retrieval tasks. Recently, inspired by the great success of Trans- former (Vaswani et al., 2017) and BERT (Devlin et al., 2019) in natural language processing, some methods (Li et al., 2019, 2020a; Chen et al., 2020; Li et al., 2020b; Fei et al., 2021) investigate cross- BERT model to exploit the cross-modal attention and devise several pre-training tasks. Benefiting from cross-modal attention and pre-training, they have achieved significantly higher retrieval accu- racy than their embedding-based counterparts. Nev- ertheless, the heavy text-image interaction from utilizing cross-modal attention leads to an O(NM) computational complexity in encoding when cal- culating the similarities between N images and M texts. The quadratic computational complexity of cross-BERT methods makes them not suitable for large-scale cross-modal retrieval applications. Several methods attempt to gain satisfactory ef- ficiency and maintain high accuracy through trad- ing off efficiency and accuracy. These methods can be coarsely grouped into two categories: two- stage methods (Sun et al., 2021; Geigle …

Content truncated to 500 words