Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5939–5958 July 9-14, 2023 ©2023 Association for Computational Linguistics Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training Zejun Li1, Zhihao Fan1, JingJing Chen2, Qi Zhang2, Xuanjing Huang2,3, Zhongyu Wei1,4∗ 1School of Data Science, Fudan University, China 2School of Computer Science, Fudan University, China 3Shanghai Collaborative Innovation Center of Intelligent Visual Computing, China 4Research Institute of Intelligent and Complex Systems, Fudan University, China {zejunli20,fanzh18,chenjingjing,qz,xjhuang,zywei}@fudan.edu.cn Abstract Multilingual Vision-Language Pre-training (VLP) is a promising but challenging topic due to the lack of large-scale multilingual image- text pairs. Existing works address the prob- lem by translating English data into other lan- guages, which is intuitive and the generated data is usually limited in form and scale. In this paper, we explore a more practical and scal- able setting: weakly supervised multilingual VLP with only English image-text pairs and multilingual text corpora. We argue that the universal multilingual representation learned from texts allows the cross-modal interaction learned in English to be transferable to other languages. To this end, we propose a frame- work to effectively unify cross-lingual and cross-modal pre-training. For unified model- ing on different data, we design an architec- ture with flexible modules to learn different interactions. Moreover, two unified tasks are introduced to efficiently guide the unified cross- lingual cross-modal learning. Extensive experi- ments demonstrate that our pre-trained model learns universal multilingual multimodal rep- resentations, allowing effective cross-lingual transfer on multimodal tasks. Code and mod- els are available at https://github.com/ FudanDISC/weakly-supervised-mVLP. 1 Introduction In recent years, self-supervised pre-training tech- nology has been studied extensively in various fields. The pre-trained models are able to en- code generalized contextual representations for texts (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020), images (Bao et al., 2021a; He et al., 2022), and image-text pairs (Chen et al., 2020; Li et al., 2020, 2021a), further facilitating the down- stream tasks and research. However, most pre- training studies are limited to English corpora. In order to overcome the language barrier and benefit ∗Corresponding author. English Image- Text Pairs Multilingual Unlabeled Text Parallel Translation Sentence Pairs (a) Publicly Open Resources Augmented Multilingual Data Multilingual VL Pre-training Unified Modeling Multilingual Unified Pre-training (b) Existing Works (c) Our Method Figure 1: An illustration of the differences between existing multilingual VLP methods and our method. a wider range of audience, it is important to extend the success of English-centric research to a multi- lingual scenario. Recent works have demonstrated the effectiveness of cross-lingual language model- ing (Conneau and Lample, 2019; Conneau et al., 2020). Based on large-scale multilingual corpora, the models are able to learn universal representa- tions for texts in multiple languages. However, large-scale and tightly associated mul- tilingual image-text pairs are unavailable and costly to acquire. Therefore, it is not straightforward to transfer existing VLP methods to other languages. As shown in Figure 1 (a, b), previous works (Ni et al., 2021; Zhou et al., 2021) …

Content truncated to 500 words