Efficient Multilingual Multi-modal Pre-training through Triple Contrastive Loss

Proceedings of the 29th International Conference on Computational Linguistics, pages 5730–5744 October 12–17, 2022. 5730 Efficient Multilingual Multi-modal Pre-training through Triple Contrastive Loss Youhan Lee Kakao Brain youhan.lee@kakaobrain.com Kyungtae Lim Hanbat National University ktlim@hanbat.ac.kr Woonhyuk Baek, Byungseok Roh, Saehoon Kim Kakao Brain wbaek,peter.roh,shkim@kakaobrain.com Abstract Learning visual and textual representations in the shared space from web-scale image- text pairs improves the performance of diverse vision-and-language tasks, as well as modality- specific tasks. Many attempts in this framework have been made to connect English-only texts and images, and only a few works have been proposed to extend this framework in multilin- gual settings with the help of many translation pairs. In this multilingual approach, a typical setup is to use pairs of (image and English-text) and translation pairs. The major limitation of this approach is that the learning signal of align- ing visual representation with under-resourced language representation is not strong, achiev- ing a sub-optimal performance of vision-and- language tasks. In this work, we propose a sim- ple yet effective enhancement scheme for pre- vious multilingual multi-modal representation methods by using a limited number of pairs of images and non-English texts. In specific, our scheme fine-tunes a pre-trained multilingual model by minimizing a triple contrastive loss on triplets of image and two different language texts with the same meaning, improving the connection between images and non-English texts. Experiments confirm that our enhance- ment strategy achieves performance gains in image-text retrieval, zero-shot image classifica- tion, and sentence embedding tasks. 1 Introduction Transferring visual representations learned from a large annotated set into downstream tasks of in- terest (Deng et al., 2009; Zhai et al., 2019) is the standard approach to achieve the state-of-the-art performance (Kuznetsova et al., 2020; Kolesnikov et al., 2020). However, due to the labeling cost, the scalability of this approach is rather limited. In contrast, self-supervised learning with contrastive losses (He et al., 2020; Chen et al., 2020) has proven to learn semantic representations without using the explicit labels, which becomes a promis- ing solution to obtaining general visual represen- tations. In addition, this paradigm combined with billion-scale unlabeled samples is competitive with the annotation-based transfer learning approaches in multiple tasks (Goyal et al., 2021). In natural language processing (NLP), pre- training with billions of corpus and transferring to downstream tasks has likewise achieved tremen- dous success. And it has been a de facto standard for a recent decade (Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020). Witnessing suc- cesses in two domains, researchers have been ac- tively attempting to find visual-and-language (VL) representations by combining supervisions that come from billions of both images and languages. (Li et al., 2019; Lu et al., 2019, 2020; Kim et al., 2021) have attempted to utilize highly curated VL datasets such as MS-COCO (Lin et al., 2014) or Visual Genome (Krishna et al., 2016). However, the non-trivial collection process is the major limi- tation in scaling up these datasets. CLIP (Radford et al., 2021) overcame the limitation by learning VL …

Content truncated to 500 words