VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-words

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5020–5029 August 1–6, 2021. ©2021 Association for Computational Linguistics 5020 VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-words Xiaopeng Lu∗ Language Technologies Institute Carnegie Mellon University xiaopen2@andrew.cmu.edu Tiancheng Zhao, Kyusong Lee SOCO Inc Pittsburgh, USA {tianchez,kyusongl}@soco.ai Abstract Text-to-image retrieval is an essential task in cross-modal information retrieval, i.e., retriev- ing relevant images from a large and unla- belled dataset given textual queries. In this pa- per, we propose VisualSparta, a novel (Visual- text Sparse Transformer Matching) model that shows significant improvement in terms of both accuracy and efficiency. VisualSparta is capable of outperforming previous state- of-the-art scalable methods in MSCOCO and Flickr30K. We also show that it achieves sub- stantial retrieving speed advantages, i.e., for a 1 million image index, VisualSparta using CPU gets ∼391X speedup compared to CPU vector search and ∼5.4X speedup compared to vector search with GPU acceleration. Ex- periments show that this speed advantage even gets bigger for larger datasets because Visu- alSparta can be efficiently implemented as an inverted index. To the best of our knowledge, VisualSparta is the first transformer-based text- to-image retrieval model that can achieve real- time searching for large-scale datasets, with significant accuracy improvement compared to previous state-of-the-art methods. 1 Introduction Text-to-image retrieval is the task of retrieving a list of relevant images from a corpus given text queries. This task is challenging because in order to find the most relevant images given text query, the model needs to not only have good representations for both textual and visual modalities, but also capture the fine-grained interaction between them. Existing text-to-image retrieval models can be broadly divided into two categories: query-agnostic and query-dependent models. The dual-encoder architecture is a common query-agnostic model, which uses two encoders to encode the query ∗ This work was partially done during an internship at SOCO Figure 1: Inference Time vs. Model Accuracy. Each dot represents Recall@1 for different models on MSCOCO 1K split. By setting top n-terms to 500, our model significantly outperforms the previous best query-agnostic retrieval models, with ∼2.8X speedup. See section 5.1 for details. and images separately and then compute the rel- evancy via inner product (Faghri et al., 2017; Lee et al., 2018; Wang et al., 2019a). The transformer architecture is a well-known query- dependent model (Devlin et al., 2018; Yang et al., 2019). In this case, each pair of text and image is encoded by concatenating and passing into one single network, instead of being encoded by two separate encoders (Lu et al., 2020; Li et al., 2020b). This method borrows the knowledge from large pre- trained transformer models and shows much better accuracy compared to dual-encoder methods (Li et al., 2020b). Besides improving the accuracy, retrieval speed has also been a long-existing subject of study in the information retrieval (IR) community (Man- ning et al., 2008). Query-dependent models are prohibitively slow to apply …

Content truncated to 500 words