Cross-lingual Visual Pre-training for Multimodal Machine Translation

Cross-lingual Visual Pre-training for Multimodal Machine Translation Ozan Caglayan1, Menekse Kuyu2, Mustafa Sercan Amac2, Pranava Madhyastha1 Erkut Erdem2, Aykut Erdem3 and Lucia Specia1,4,5 Imperial College London1, Hacettepe University2, Koc¸ University3 University of Sheffield4, ADAPT - Dublin City University5 o.caglayan@ic.ac.uk, meneksekuyu@gmail.com, sercanamac@gmail.com, pranava@ic.ac.uk erkut@cs.hacettepe.edu.tr, aerdem@ku.edu.tr, l.specia@ic.ac.uk Abstract Pre-trained language models have been shown to improve performance in many natural lan- guage tasks substantially. Although the early focus of such models was single language pre-training, recent advances have resulted in cross-lingual and visual pre-training meth- ods. In this paper, we combine these two approaches to learn visually-grounded cross- lingual representations. Specifically, we ex- tend the translation language modelling (Lam- ple and Conneau, 2019) with masked region classification and perform pre-training with three-way parallel vision & language corpora. We show that when fine-tuned for multimodal machine translation, these models obtain state- of-the-art performance. We also provide qualitative insights into the usefulness of the learned grounded representations. 1 Introduction Pre-trained language models (Peters et al., 2018; Devlin et al., 2019) have been proven valuable tools for contextual representation extraction. Many studies have shown their effectiveness in discov- ering linguistic structures (Tenney et al., 2019), which is useful for a wide variety of NLP tasks (Tal- mor et al., 2019; Kondratyuk and Straka, 2019; Petroni et al., 2019). These positive results led to further exploration of (i) cross-lingual pre- training (Lample and Conneau, 2019; Conneau et al., 2020; Wang et al., 2020) through the use of multiple mono-lingual and parallel resources, and (ii) visual pre-training where large-scale image cap- tioning corpora are used to induce grounded vision & language representations (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2020a; Su et al., 2020; Li et al., 2020b). The latter is usually achieved by extending the masked language modelling (MLM) objective (Devlin et al., 2019) with auxiliary vision & language tasks such as masked region classifica- tion and image sentence matching. In this paper, we present the first attempt to bring together cross-lingual and visual pre-training. Our visual translation language modelling (VTLM) objective combines the translation language mod- elling (TLM) (Lample and Conneau, 2019) with masked region classification (MRC) (Chen et al., 2020; Su et al., 2020) to learn grounded cross- lingual representations. Unlike most of the prior work that use classification or retrieval based down- stream evaluation, we focus on the generative task of multimodal machine translation (MMT), where images accompany captions during translation (Su- lubacak et al., 2020). Once pre-trained, we trans- fer the VTLM encoder to a Transformer-based (Vaswani et al., 2017) MMT and fine-tune it for the MMT task. To our knowledge, this is also the first attempt of pre-training & fine-tuning for MMT, where the current state of the art mostly relies on training multimodal sequence-to-sequence systems from scratch (Calixto et al., 2016; Caglayan et al., 2016; Libovick´y and Helcl, 2017; Elliott and K´ad´ar, 2017; Caglayan et al., 2017; Yin et al., 2020). Our findings highlight the effectiveness of cross- lingual visual pre-training: when fine-tuned on the English→German …

Content truncated to 500 words