Probing Multi-modal Machine Translation with Pre-trained Language Model

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3689–3699 August 1–6, 2021. ©2021 Association for Computational Linguistics 3689 Probing Multi-modal Machine Translation with Pre-trained Language Model Yawei Kong, Kai Fan Alibaba DAMO Academy {yawei.kyw,k.fan}@alibaba-inc.com Abstract Multi-modal machine translation (MMT) aimed at using images to help disambiguate the target during translation and improving robustness, but some recent works showed that the contribution of visual features is either negligible or incremental. In this paper, we show that incorporating pre-trained (vision) language model (VLP) on the source side can improve the multi-modal translation quality significantly. Motivated by BERT, VLP aims to learn better cross-modal representations that improve target sequence generation. We simply adapt BERT to a cross-modal domain for the vision language pre-training, and the downstream multi-modal machine translation can substantially benefit from the pre-training. We also introduce an attention based modality loss to promote the image-text alignment in the latent semantic space. Ablation study verifies that it is effective in further improving the translation quality. Our experiments on the widely used Multi-30K dataset show increased BLEU score up to 6.2 points compared with the text-only model, achieving inthe state-of-the-art results with a large margin the semi-unconstrained scenario and indicating a possible direction to rejuvenate the multi-modal machine translation. 1 Introduction Joint models of language and vision have achieved remarkable results, such as in image caption (Karpathy and Fei-Fei, 2015) and visual question answering (Antol et al., 2015). Multi-modal ma- chine translation (MMT) was first introduced as a shared competition task at the 2016 Conference on Machine Translation (WMT16) (Specia et al., 2016) as an interdisciplinary study to incorporate a visual element into the multilingual translation task. This task continued for three years until WMT18, and the findings presented by the organizers sug- gest that the text-only systems remain competi- tive, and that the contribution of visual modality Data used img src tgt examples Multi-30K ✓ ✓ ✓ most works +external data ✓ ✓ (Gr¨onroos et al., 2018) ✓ ✓ (Helcl et al., 2018) ✓ ✓ (Yin et al., 2020), ours Table 1: Different unconstrained scenarios in MMT. is not entirely convincing (Specia et al., 2016; El- liott et al., 2017; Barrault et al., 2018). Moreover, the experiments in (Elliott, 2018) find that a pub- licly available MMT system produces great trans- lations with random, incongruent images, further undermining the importance of visual features. The empirical results have so far raised doubts about whether the visual features can really help MMT, and there is evidence pointing to a negative answer. We hypothesize that one reason is the data scale of the benchmarking Multi-30K (Elliott et al., 2016) – it is likely insufficient for a deep model to learn better cross-modality or cross-lingual repre- sentations. However, the pre-training techniques such as BERT (Devlin et al., 2019) or cross-lingual language model (XLM) (Conneau and Lample, 2019) can capture rich representations of the inputs from languages and be applied to various down- stream tasks by providing context-aware embed- dings, leading to remarkable improvements even …

Content truncated to 500 words