Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2443–2459 June 6–11, 2021. ©2021 Association for Computational Linguistics 2443 Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models Po-Yao Huang13∗, Mandela Patrick23∗, Junjie Hu1, Graham Neubig1, Florian Metze3, Alexander Hauptmann1 1School of Computer Science, Carnegie Mellon University 2Visual Geometry Group, University of Oxford 3Facebook AI {poyaoh,junjieh,gneubig,alex}@cs.cmu.edu, {mandelapatrick,fmetze}@fb.com Abstract This paper studies zero-shot cross-lingual transfer of vision-language models. Specif- ically, we focus on multilingual text-to- video search and propose a Transformer-based model that learns contextual multilingual mul- timodal embeddings. Under a zero-shot set- ting, we empirically demonstrate that perfor- mance degrades significantly when we query the multilingual text-video model with non- English sentences. To address this prob- lem, we introduce a multilingual multimodal pre-training strategy, and collect a new mul- tilingual instructional video dataset (Multi- HowTo100M) for pre-training. Experiments on VTT show that our method significantly im- proves video search in non-English languages without additional annotations. Furthermore, when multilingual annotations are available, our method outperforms recent baselines by a large margin in multilingual text-to-video search on VTT and VATEX; as well as in mul- tilingual text-to-image search on Multi30K. Our model and Multi-HowTo100M is avail- able at http://github.com/berniebear/ Multi-HT100M 1 Introduction One of the key challenges at the intersection of computer vision (CV) and natural language pro- cessing (NLP) is building versatile vision-language models that not only work in English, but in all of the world’s approximately 7,000 languages. Since collecting and annotating task-specific parallel mul- timodal data in all languages is impractical, a framework that makes vision-language models gen- eralize across languages is highly desirable. One technique that has shown promise to greatly improve the applicability of NLP models to new languages is zero-shot cross-lingual transfer, where models trained on a source language are applied ∗Equal contribution. as-is to a different language without any additional annotated training data (T¨ackstr¨om et al., 2012; Klementiev et al., 2012; Cotterell and Heigold, 2017; Chen et al., 2018; Neubig and Hu, 2018). In particular, recent techniques for cross-lingual trans- fer have demonstrated that by performing unsuper- vised learning of language or translation models on many languages, followed by downstream task fine-tuning using only English annotation, models can nonetheless generalize to a non-English lan- guage (Wu and Dredze, 2019a; Lample and Con- neau, 2019; Huang et al., 2019a; Artetxe et al., 2020; Hu et al., 2020). This success is attributed to the fact that many languages share a considerable amount of underlying vocabulary or structure. At the vocabulary level, languages often have words that stem from the same origin, for instance, “desk” in English and “Tisch” in German both come from the Latin “discus”. At the structural level, all lan- guages have a recursive structure, and many share traits of morphology or word order. For cross-lingual transfer of vision-language models, the visual information is clearly an essen- tial element. To this end, we make an important yet under-explored step to incorporate visual-textual re- …

Content truncated to 500 words