LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 982–997 June 6–11, 2021. ©2021 Association for Computational Linguistics 982 LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval Siqi Sun∗, Yen-Chun Chen∗, Linjie Li, Shuohang Wang, Yuwei Fang, Jingjing Liu Microsoft Corporation {siqi.sun, yen-chun.chen, lindsey.li, shuohang.wang, yuwfan, jingjl}@microsoft.com Abstract Multimodal pre-training has propelled great advancement in vision-and-language research. These large-scale pre-trained models, although successful, fatefully suffer from slow infer- ence speed due to enormous computation cost mainly from cross-modal attention in Trans- former architecture. When applied to real- life applications, such latency and computa- tion demand severely deter the practical use of pre-trained models. In this paper, we study Image-text retrieval (ITR), the most ma- ture scenario of V+L application, which has been widely studied even prior to the emer- gence of recent pre-trained models. We pro- pose a simple yet highly effective approach, LightningDOT that accelerates the inference time of ITR by thousands of times, with- out sacrificing accuracy. LightningDOT re- moves the time-consuming cross-modal atten- tion by pre-training on three novel learning objectives, extracting feature indexes offline, and employing instant dot-product matching with further re-ranking, which significantly speeds up retrieval process. In fact, Light- ningDOT achieves new state of the art across multiple ITR benchmarks such as Flickr30k, COCO and Multi30K, outperforming existing pre-trained models that consume 1000× mag- nitude of computational hours.1 1 Introduction Image-text retrieval (ITR) has been widely stud- ied as a staple benchmark task in both NLP and computer vision communities. Traditional ITR search engines typically deploy ranking-based mod- els built upon visual-semantic embedding match- ing (Faghri et al., 2017; Huang et al., 2018) or deep cross-modal fusion with attention mecha- nism (Lee et al., 2018; Li et al., 2020a,b). Ear- liest works (Kiros et al., 2014; Faghri et al., 2017; ∗Equal Contribution. 1Code and pre-training checkpoints are available at https://github.com/intersun/LightningDOT. MLM /MRM/ITM/… 1. CNN fo product • CNN 2. Cross at LSTM o 3. Pre-trai Unicode • Self-at • Deep t 4. 1.Propose Co acc … RNN Faster -RCNN CNN RNN Transformer … Transformer Dot Product Word Features Region Features Global Text Features Global Image Features Cross Attention [CLS] Features Transformer SMRM VMLM CMR (a) (b) (c) (d) Figure 1: Evolution of Image-Text Retrieval (ITR) paradigm. (a) Early work (Faghri et al., 2017) using dot product to learn the similarity between global image features and global text features. (b) Later study (Lee et al., 2018) applying cross-attention between the features of each region and each word. (c) Pre-trained V+L models (Chen et al., 2020) with deep Transformer. (d) LightningDOT without cross-attention. CMR, SMRM and VMLM refer to different pre-training tasks, which will be introduced later in method section. Wang et al., 2018) employ separate image en- coder (e.g., CNN) and text encoder (e.g., RNN), the embeddings from which are then measured by doc product for similarity matching (Figure 1(a)). Later studies (Lee et al., 2018, 2019; Wang et al., 2019; Zhang et al., …

Content truncated to 500 words