Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6153–6166 August 1–6, 2021. ©2021 Association for Computational Linguistics 6153 Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation Zhiyong Wu† ∗, Lingpeng Kong†¶, Wei Bi‡, Xiang Li§, Ben Kao† †The University of Hong Kong, ‡Tencent AI Lab, ¶Shanghai Artificial Intelligence Laboratory, §East China Normal University †{zywu,lpk,kao}@cs.hku.hk, ‡victoriabi@tencent.com, §xiangli@dase.ecnu.edu.cn Abstract A neural multimodal machine translation (MMT) system is one that aims to perform bet- ter translation by extending conventional text- only translation models with multimodal infor- mation. Many recent studies report improve- ments when equipping their models with the multimodal module, despite the controversy of whether such improvements indeed come from the multimodal part. We revisit the contribu- tion of multimodal information in MMT by de- vising two interpretable MMT models. To our surprise, although our models replicate sim- ilar gains as recently developed multimodal- integrated systems achieved, our models learn to ignore the multimodal information. Upon further investigation, we discover that the im- provements achieved by the multimodal mod- els over text-only counterparts are in fact re- sults of the regularization effect. We report em- pirical findings that highlight the importance of MMT models’ interpretability, and discuss how our findings will benefit future research. 1 Introduction Multimodal Machine Translation (MMT) aims at designing better translation systems by extending conventional text-only translation systems to take into account multimodal information, especially from visual modality (Specia et al., 2016; Wang et al., 2019). Despite many previous success in MMT that report improvements when models are equipped with visual information (Calixto et al., 2017; Helcl et al., 2018; Ive et al., 2019; Lin et al., 2020; Yin et al., 2020), there have been continuing debates on the need for visual context in MMT. In particular, Specia et al. (2016); Elliott et al. (2017); Barrault et al. (2018) argue that visual con- text does not seem to help translation reliably, at ∗The majority of this work was done while the first author was interning at Tencent AI Lab. least as measured by automatic metrics. Elliott (2018); Gr¨onroos et al. (2018a) provide further ev- idence by showing that MMT models are, in fact, insensitive to visual input and can translate without significant performance losses even in the pres- ence of features derived from unrelated images. A more recent study (Caglayan et al., 2019), however, shows that under limited textual context (e.g., noun words are masked), models can leverage visual in- put to generate better translations. But it remains unclear where the gains of MMT methods come from, when the textual context is complete. The main tool utilized in prior discussion is ad- versarial model comparison — explaining the be- havior of complex and black-box MMT models by comparing performance changes when given adver- sarial input (e.g., random images). Although such an opaque tool is an acceptable beginning to in- vestigate the need for visual …

Content truncated to 500 words