Leaner and Faster: Two-Stage Model Compression for Lightweight Text-Image Retrieval

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4085 - 4090 July 10-15, 2022 ©2022 Association for Computational Linguistics Leaner and Faster: Two-Stage Model Compression for Lightweight Text-Image Retrieval Siyu Ren Kenny Q. Zhu∗ Shanghai Jiao Tong University Shanghai, China roy0702@sjtu.edu.cn, kzhu@cs.sjtu.edu.cn Abstract Current text-image approaches (e.g., CLIP) typically adopt dual-encoder architecture us- ing pre-trained vision-language representation. However, these models still pose non-trivial memory requirements and substantial incre- mental indexing time, which makes them less practical on mobile devices. In this paper, we present an effective two-stage framework to compress large pre-trained dual-encoder for lightweight text-image retrieval. The result- ing model is smaller (39% of the original), faster (1.6x/2.9x for processing image/text re- spectively), yet performs on par with or bet- ter than the original full model on Flickr30K and MSCOCO benchmarks. We also open- source an accompanying realistic mobile im- age search application. 1 Introduction Text-image retrieval is the task aiming at retriev- ing a list of relevant images from a large set of images given a textual query specified by the user. Recently, large-scale vision-language pre- training (VLP) has spawned models (Tan and Bansal, 2019; Li et al., 2020; Radford et al., 2021) that established state-of-the-art results in various vision-language tasks (Antol et al., 2015; Suhr et al., 2019), including text-image retrieval. Ex- isting VLP models for text-image retrieval can be divided into two categories: cross-encoder architec- ture and dual-encoder architecture. Cross-encoder models show better retrieval accuracy by allowing fine-grained cross-modal attention among image and text. However, they are prohibitively slow to apply to the entire image pool because each im- age has to go through the deep Transformer again whenever a new text query comes in. Moreover, most cross-encoder models rely on external object detection models (Ren et al., 2015) to extract visual ∗ The corresponding author. features, which further increase memory consump- tion. On the other hand, dual-encoder models are more scalable in that they allow pre-computing im- age representations as reusable vectors independent of the text queries. These image vectors can be in- dexed and efficiently retrieved at runtime using Ap- proximate Nearest Neighbor (ANN) search (John- son et al., 2017). As long as the image pool remains unchanged, the image encoder is not required. However, a more practical scenario calls for dy- namic indexing of new images into the pool (e.g., private photo collections on mobile devices), which requires both the image encoder and the text en- coder to be resident in memory. This makes the above approach less practical on mobile devices with limited memory and processing power. Un- fortunately, little attention has been paid to fulfill this need. In this paper, we show that a large dual- encoder model can be compressed into a much smaller and faster counterpart while retaining its retrieval accuracy using a novel two-stage com- pression framework. In the first stage, we make use of abundant non-paired texts/images to sep- arately compress text or image encoder …

Content truncated to 500 words