Hashing based Efficient Inference for Image-Text Matching

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 743–752 August 1–6, 2021. ©2021 Association for Computational Linguistics 743 Hashing based Efficient Inference for Image-Text Matching Rong-Cheng Tu†, Lei Ji ‡§¶∗, Huaishao Luo∥, Botian Shi≀, Heyan Huang†, Nan Duan¶ and Xian-Ling Mao† †School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China, Beijing, China ‡Institute of Computing Technology, CAS, Beijing, China §University of Chinese Academy of Sciences, Beijing, China ¶Microsoft Research Asia, Beijing, China ∥Southwest Jiaotong University, Chengdu, China ≀Shanghai AI lab, Shanghai, China †{turongcheng, hhy63, maoxl}@bit.edu.cn, ¶{leiji,nanduan}@microsoft.com Abstract Image-text matching has been a popular re- search topic which bridges vision and lan- guage through semantic understanding. Re- cent works mainly focus on exploring the in- teractions between images and sentences to im- prove the performance without considering in- ference efficiency. Specifically, for the large scale databases, it is unacceptable to perform such time-consuming mechanisms between a query (text/image) and each candidate data- point (image/text) in the whole retrieval set during inference. To tackle this problem, we propose a novel hashing based efficient infer- ence module called HEI, which can be plugged into the existing framework to speed up the inference step without reducing the retrieval performance. In details, HEI learns to map the original datapoints into short binary hash codes and coarsely preserve the heterologous matching relationship. Thus, in the infer- ence phase, the proposed HEI module uses the hash codes to quickly select a few can- didate datapoints from the retrieval set for a given query. Then, the image-text matching model fine ranks the candidate set to find the matching datapoint. Extensive experiments on two widely used benchmark MS-COCO and Flickr30k with four baseline methods demon- strate the efficiency and effectiveness of our proposed HEI module. 1 Introduction Language and vision understanding plays a fun- damental role for human to understand the real world. A large amount of works has been proposed to bridge these two modalities. Image-text match- ing is one of the fundamental topics in this field, which benefits a series of downstream applications, such as visual captioning(Zhang et al., 2019; Wang et al., 2018) and visual grounding (Chen et al., ∗Corresponding author. 2018; Plummer et al., 2017). Specifically, given an image (text), its target is to match the closest textual description (image) for the image (text). Early works (Karpathy and Fei-Fei, 2015; Wang et al., 2016; Niu et al., 2017; Faghri et al., 2017) achieve this goal by learning two modality-specific deep neural networks to directly map all the data- points from the two modality into a common joint space without using attention mechanism, and then measures their similarities by feature distances in the joint space. Compared with these methods, recent works (Lee et al., 2018; Liu et al., 2019; Wang et al., 2019; Chen et al., 2020) mainly fo- cus on incorporating variant attention mechanisms into the image-text matching models to explore the fine-grained interactions between vision and language. By using the attention mechanisms, the image-text matching models are able to …

Content truncated to 500 words