LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2862 - 2872 December 7-11, 2022 ©2022 Association for Computational Linguistics LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation Hongcheng Guo*1, Jiaheng Liu*1, Haoyang Huang2, Jian Yang1, Zhoujun Li�1, Dongdong Zhang2 , Zheng Cui2 1Beihang University 2Microsoft Research Asia {hongchengguo,liujiaheng,jiaya,lizj}@buaa.edu.cn {haohua,dozhang,zhcui}@microsoft.com Abstract Multimodal Machine Translation (MMT) fo- cuses on enhancing text-only translation with visual features, which has attracted consider- able attention from both natural language pro- cessing and computer vision communities. Re- cent advances still struggle to train a sepa- rate model for each language pair, which is costly and unaffordable when the number of languages increases in the real world. In other words, the multilingual multimodal machine translation (Multilingual MMT) task has not been investigated, which aims to handle the aforementioned issues by providing a shared semantic space for multiple languages. Be- sides, the image modality has no language boundaries, which is superior to bridging the semantic gap between languages. To this end, we first propose the Multilingual MMT task by establishing two new Multilingual MMT benchmark datasets covering seven languages. Then, an effective baseline LVP-M3 using vi- sual prompts is proposed to support translations between different languages, which includes three stages (token encoding, language-aware visual prompt generation, and language trans- lation). Extensive experimental results on our constructed benchmark datasets demonstrate the effectiveness of LVP-M3 method for Multi- lingual MMT. 1 Introduction Multimodal Machine Translation (MMT) extends the conventional text-based machine translation by taking corresponding images as additional in- puts (Lin et al., 2020; Li et al., 2022) to mitigate the problems of data sparsity and ambiguity (Ive et al., 2019; Yang et al., 2022) when compared with purely text-based machine translation. Simi- lar to other multimodal tasks (e.g., visual question answering (Antol et al., 2015; Shih et al., 2016), image captioning (Vinyals et al., 2015; Jia et al., * First two authors contributed equally. � Corresponding author. Target Language De: Ein Junge taucht in... MMT Model (En-De) MMT Model (En-Fr) MMT Model (En-Cs) Target Language De: Ein Junge taucht in... Fr: Un garc¸on plonge... Cs: Chlapec ska´ce do... ... Multilingual MMT Model Fr: Un garc¸on plonge... Cs: Chlapec ska´ce do... En: A boy dives into... Source Language Image (a) MMT (b) Multilingual MMT En: A boy dives into... Source Language Image Figure 1: Comparison of MMT and Multilingual MMT. (a) For MMT, we need to train different MMT models to support translations between different language pairs (e.g., “En-De” represents to translate the English to German). (b). For Multilingual MMT, we only need one single model to translate the source language to different target languages. 2015) and video-text retrieval (Liu et al., 2022d)), MMT aims to exploit the effectiveness of vision information for the machine translation task. Moreover, MMT has broad applications (Zhou et al., 2018), such as multimedia news and movie subtitles in different languages. However, as shown in Fig. 1(a), previous MMT models (e.g., DCCN (Lin et al., 2020)) can handle a single …

Content truncated to 500 words