Cross-lingual Cross-modal Pretraining for Multimodal Retrieval

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3644–3650 June 6–11, 2021. ©2021 Association for Computational Linguistics 3644 Cross-lingual Cross-modal Pretraining for Multimodal Retrieval Hongliang Fei, Tan Yu, Ping Li Cognitive Computing Lab Baidu Research 10900 NE 8th St, Bellevue, WA 98004, USA {hongliangfei, tanyu01, liping11}@baidu.com Abstract Recent pretrained vision-language models have achieved impressive performance on cross-modal retrieval tasks in English. Their success, however, heavily depends on the availability of many annotated image-caption datasets for pretraining, where the texts are not necessarily in English. Although we can uti- lize machine translation (MT) tools to trans- late non-English text to English, the perfor- mance still largely relies on MT’s quality and may suffer from high latency problems in real- world applications. This paper proposes a new approach to learn cross-lingual cross-modal representations for matching images and their relevant captions in multiple languages. We seamlessly combine cross-lingual pretraining objectives and cross-modal pretraining objec- tives in a unified framework to learn image and text in a joint embedding space from available English image-caption data, monolingual and parallel corpus. We show that our approach achieves SOTA performance in retrieval tasks on two multimodal multilingual image caption benchmarks: Multi30k with German captions and MSCOCO with Japanese captions. 1 Introduction Recent pretrained vision-language models (Chen et al., 2020; Li et al., 2020; Su et al., 2020; Gan et al., 2020; Luo et al., 2020) based on Trans- former (Vaswani et al., 2017) have achieved re- markable performance on cross-modal retrieval (Li et al., 2020; Yu et al., 2020, 2021b), image cap- tioning (Chen et al., 2020) and visual question and answering (VQA) (Su et al., 2020) tasks in English. For instance, most leading competitors in the VQA contest1 rely on the transformer-based pretrained vision-language models. However, their success heavily depends on the availability of a large amount of annotated image- caption pretraining datasets (e.g., conceptual cap- 1https://visualqa.org/roe.html tions (Sharma et al., 2018)). In reality, there are limited such data in other languages. When gen- eralizing to cross-lingual cross-modal downstream tasks, a straightforward way is to utilize machine translation (MT) tools to translate non-English text to English and reuse the fine-tuned models in En- glish. Nevertheless, the performance strongly relies on the MT tool’s capability and suffers from high latency problems in real-world applications. To learn multilingual multimodal representa- tions, recent researchers utilized multilingual datasets to model images and text captions in a joint embedding space. Based on how the shared feature space is learned, there are two categories: word- level alignments (Mohammadshahi et al., 2019) and sentence-level alignments (Wehrmann et al., 2019; Rajendran et al., 2016). Those models can capture a certain level of semantic similarity among languages and images. They, however, only mod- eled the relevance of text and images in a global manner. Such a limitation may prevent these mod- els from effectively detecting relevance locally. In parallel, cross-lingual language models such as multilingual BERT (Devlin et al., 2019) and XLM (Conneau …

Content truncated to 500 words