Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8556–8562 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8556 Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models Jiaoda Li1 Duygu Ataman2∗ Rico Sennrich3,4 1ETH Zürich 2New York University 3University of Zürich jiaoda.li@inf.ethz.ch4University of Edinburgh ataman@nyu.edu sennrich@cl.uzh.ch Abstract Multimodal machine translation (MMT) sys- tems have been shown to outperform their text- only neural machine translation (NMT) coun- terparts when visual context is available. How- ever, recent studies have also shown that the per- formance of MMT models is only marginally impacted when the associated image is replaced with an unrelated image or noise, which sug- gests that the visual context might not be ex- ploited by the model at all. We hypothesize that this might be caused by the nature of the com- monly used evaluation benchmark, also known as Multi30K, where the translations of image captions were prepared without actually show- ing the images to human translators. In this paper, we present a qualitative study that ex- amines the role of datasets in stimulating the leverage of visual modality and we propose methods to highlight the importance of visual signals in the datasets which demonstrate im- provements in reliance of models on the source images. Our findings suggest the research on effective MMT architectures is currently im- paired by the lack of suitable datasets and care- ful consideration must be taken in creation of future MMT datasets, for which we also pro- vide useful insights.1 1 Introduction Multimodal machine translation (MMT) aims to improve machine translation by resolving certain contextual ambiguities with the aid of other modal- ities such as vision, and have shown promising integration in conventional neural machine trans- lation (NMT) models (Specia et al., 2016). On the other hand, recent studies reported some con- flicting results regarding how the additional visual information is exploited by the models for generat- ing higher-quality translations. A number of MMT ∗ Work done while at the University of Zürich. 1Our code and data are available at: https://github.com/jiaodali/ vision-matters-when-it-should. models (Calixto et al., 2017; Helcl et al., 2018; Ive et al., 2019; Lin et al., 2020; Yin et al., 2020) have been proposed which showed improvements over text-only models, whereas Lala et al. (2018); Bar- rault et al. (2018); Raunak et al. (2019) observed that the multimodal integration did not make a big difference quantitatively or qualitatively. Follow- ing experimental work showed that replacing the images in image-caption pairs with incongruent im- ages (Elliott, 2018) or even random noise (Wu et al., 2021) might still result in similar performance of multimodal models. In light of these results, Wu et al. (2021) suggested that gains in quality might merely be due to a regularization effect and the images may not actually be exploited by models during the translation task. In this paper, we investigate the role of the evaluation benchmark in model performance and whether its tendency to ignore visual information in the input could be …

Content truncated to 500 words