Make the Blind Translator See The World: A Novel Transfer Learning Solution for Multimodal Machine Translation

Make the Blind Translator See The World: A Novel Transfer Learning Solution for Multimodal Machine Translation Minghan Wang wangminghan@huawei.com Jiaxin Guo guojiaxin1@huawei.com Yimeng Chen chenyimeng@huawei.com Chang Su suchang8@huawei.com Min Zhang zhangmin186@huawei.com Shimin Tao taoshimin@huawei.com Hao Yang yanghao30@huawei.com Huawei Translation Service Center, Beijing, China Abstract Based on large-scale pretrained networks, the liability to be easily overfitting with limited la- belled training data of multimodal translation (MMT) is a critical issue in MMT. To this end, we propose a transfer learning solution. Specifically, 1) A vanilla Transformer is pre-trained on massive bilingual text-only corpus to obtain prior knowledge; 2) A multimodal Transformer named VLTransformer is proposed with several components incorporated visual contexts; and 3) The parameters of VLTransformer are initialized with the pre-trained vanilla Transformer, then being fine-tuned on MMT tasks with a newly proposed method named cross-modal mask- ing which forces the model to learn from both modalities. We evaluated on the Multi30k en-de and en-fr dataset, improving up to 8% BLEU score compared with the SOTA performance. The experimental result demonstrates that performing transfer learning with monomodal pre- trained NMT model on multimodal NMT tasks can obtain considerable boosts. 1 Introduction Transformer-based models using large-scale parallel corpora have significantly improved the performance of neural machine translation (NMT), marking an important milestone (Vaswani et al., 2017). Additionally, multimodal machine translation (MMT) incorporating image sig- nals into RNN-based encoder-decoder shows improvements on translation quality due to the forceful disambiguation (Specia et al., 2016a). In this paper, we aim to investigate, on top of Transformer, whether the paradigm of first pretraining and then fine-tuning can be effectively applied to MMT, concretely transferring from monomodal to multimodal tasks. Constant attention has been paid on MMT task (Specia et al., 2016a) in the Conference of Machine Translation (WMT) in recent years (2016-2018). Formally, it aims to learn a function mapping: X × I → Y, which takes source text and an image as input and translate them into the target text as shown in Figure 1. Additional modality is to disambiguate the source sentence, with the reference of image. However, the effectiveness of the visual context has been ques- tioned by prior work (Specia et al., 2016b; Elliott et al., 2017; Barrault et al., 2018; Caglayan et al., 2019). They show that visual context is not convincingly useful and the marginal gain is pretty modest, which is speculated to be resulted from the limitation of available datasets — the Proceedings of the 18th Biennial Machine Translation Summit Virtual USA, August 16 - 20, 2021, Volume 1: MT Research Track Page 139 Figure 1: An example of the multimodal translation. (Specia et al., 2016b) Large-scale text-only bilingual corpus Limited multimodal bilingual corpus Pre-Training Fine-Tuning + Vision Text + Image Figure 2: The multimodal transfer learning solution. 1) Initialize a vanilla Transformer. 2) Train the model with large-scale parallel corpus. 3) Add visual related components. 4) Fine-tune the model on limited multimodal corpus. scale of parallel dataset of MMT task is not enough to train a robust …

Content truncated to 500 words