On Vision Features in Multimodal Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6327 - 6337 May 22-27, 2022 c⃝2022 Association for Computational Linguistics On Vision Features in Multimodal Machine Translation Bei Li1, Chuanhao Lv1, Zefan Zhou1, Tao Zhou1, Tong Xiao1,2∗, Anxiang Ma1,2 and Jingbo Zhu1,2 1School of Computer Science and Engineering, Northeastern University, Shenyang, China 2NiuTrans Research, Shenyang, China {libei_neu,lch-sy,ZhouZefan_zzf,zhoutao_neu}@outlook.com {xiaotong,maanxiang,zhujingbo}@mail.neu.edu.cn Abstract Previous work on multimodal machine trans- lation (MMT) has focused on the way of in- corporating vision features into translation but little attention is on the quality of vision mod- els. In this work, we investigate the impact of vision models on MMT. Given the fact that Transformer is becoming popular in computer vision, we experiment with various strong models (such as Vision Transformer) and en- hanced features (such as object-detection and image captioning). We develop a selective at- tention model to study the patch-level contribu- tion of an image in MMT. On detailed probing tasks, we find that stronger vision models are helpful for learning translation from the visual modality. Our results also suggest the need of carefully examining MMT models, especially when current benchmarks are small-scale and biased. Our code could be found at https: //github.com/libeineu/fairseq_mmt. 1 Introduction Multimodal machine translation (MMT) has emerged as an active field of research which mar- ries the worlds of computer vision (CV) and natural language processing (NLP) (Specia et al., 2016). Early models of this kind produce a translation given the fused representation of both the visual and textual inputs (Caglayan et al., 2016; Libovický and Helcl, 2017; Calixto and Liu, 2017). As ex- pected, such a paradigm achieves promising BLEU improvements and inspires the community to fol- low up. But soon researchers found that MMT systems did not act as what they ordinarily designed: the visual modality contributes to translation little. For example, it is not harmful to MMT systems when the input image is irrelevant to the text (Grönroos et al., 2018; Lala et al., 2018), or even when the vision features are absent (Elliott, 2018). More re- cently, Wu et al. (2021) have pointed out that the ∗Corresponding author. use of the visual modality is a way of regulariza- tion for training but not a complement to the text modality. As another response to the analysis of MMT, Caglayan et al. (2019) investigate how the vision features correlate to the text. They find that the input image helps translation when some of the input words are masked. Note that previous work has for the most part focused on integrating off-the-shelf vision models (such as ResNet-50) into MMT. The underlying assumption here is that the existing vision models are powerful enough to encode the image. This implicitly ignores the quality of vision models in representing images. But computer vision is facing a new trend by moving from CNNs to Transformer as the backbone model (Dosovitskiy et al., 2021; Liu et al., 2021b; Carion et al., 2020). A natu- ral question that arises …

Content truncated to 500 words