mCLIP: Multilingual CLIP via Cross-lingual Transfer

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13028–13043 July 9-14, 2023 ©2023 Association for Computational Linguistics mCLIP: Multilingual CLIP via Cross-lingual Transfer Guanhua Chen1, Lu Hou2, Yun Chen3, Wenliang Dai5, Lifeng Shang2, Xin Jiang2, Qun Liu2, Jia Pan4, Wenping Wang6 1Southern University of Science and Technology; 2Huawei Noah’s Ark Lab 3Shanghai University of Finance and Economics; 4The University of Hong Kong; 5The Hong Kong University of Science and Technology; 6Texas A&M University chengh3@sustech.edu.cn, yunchen@sufe.edu.cn, wdaiai@connect.ust.hk {houlu3, shang.lifeng, jiang.xin, qun.liu}@huawei.com jpan@cs.hku.hk, wenping@tamu.edu Abstract Large-scale vision-language pretrained (VLP) models like CLIP have shown remarkable per- formance on various downstream cross-modal tasks. However, they are usually biased to- wards English due to the lack of sufficient non-English image-text pairs. Existing mul- tilingual VLP methods often learn retrieval- inefficient single-stream models by translation- augmented non-English image-text pairs. In this paper, we introduce mCLIP, a retrieval- efficient dual-stream multilingual VLP model, trained by aligning the CLIP model and a Mul- tilingual Text Encoder (MTE) through a novel Triangle Cross-modal Knowledge Distillation (TriKD) method. It is parameter-efficient as only two light projectors on the top of them are updated during distillation. Furthermore, to enhance the token- and sentence-level multi- lingual representation of the MTE, we propose to train it with machine translation and con- trastive learning jointly before the TriKD to provide a better initialization. Empirical results show that mCLIP achieves new state-of-the-art performance for both zero-shot and finetuned multilingual image-text retrieval task. 1 Introduction Recently, large-scale dual-stream vision-language pretrained (VLP) models, such as CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021) and their variants (Yao et al., 2021; Mu et al., 2021; Zhai et al., 2021), have shown remarkable performance on various downstream multimodal tasks. These models use separate encoders for the images and texts, and allow efficient inference in the image-text retrieval task because the image or text features can be computed offline. However, most current VLP models are biased toward English, due to the lack of sufficient high-quality multilingual multimodal datasets for direct large-scale pretraining. Despite the lack of sufficient non-English image- text pairs, previous methods attempt to create word- level code-switched image-text pairs by looking up bilingual dictionaries (Ni et al., 2021), or sentence- level augmented multilingual image-text pairs by translating the English text to other languages (i.e, translate-train pipeline) (Zhou et al., 2021). Then each image and its paired text are concate- nated as a single sequence to train a single-stream Transformer-based model. Despite the good per- formance of these models (Ni et al., 2021; Zhou et al., 2021), they are less efficient than dual-stream models on large-scale image-text retrieval tasks, as the data from both modalities are intertwined to compute the self-attention and the unimodal fea- tures can not be pre-computed. Instead of creating word-level or sentence-level multilingual image- text pairs, MURAL (Jain et al., 2021) extends the ALIGN (Jia et al., 2021) model with multi- linguality by an additional text-text contrastive loss among hundreds of languages. However, …

Content truncated to 500 words