Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6755 - 6764 December 7-11, 2022 ©2022 Association for Computational Linguistics Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective Baijun Ji1, Tong Zhang1, Yicheng Zou2, Bojie Hu1∗, Si Shen3 1 Tencent Minority-Mandarin Translation, Beijing, China 2 School of Computer Science, Fudan University 3 Research Base on Interdisciplinary Terminology and Translation, Nanjing University 1{baijunji,zatozhang, bojiehu}@tencent.com 2 yczou18@fudan.edu.cn Abstract Multimodal machine translation (MMT) aims to improve translation quality by equipping the source sentence with its corresponding im- age. Despite the promising performance, MMT models still suffer the problem of input degra- dation: models focus more on textual informa- tion while visual information is generally over- looked. In this paper, we endeavor to improve MMT performance by increasing visual aware- ness from an information theoretic perspective. In detail, we decompose the informative visual signals into two parts: source-specific informa- tion and target-specific information. We use mutual information to quantify them and pro- pose two methods for objective optimization to better leverage visual signals. Experiments on two datasets demonstrate that our approach can effectively enhance the visual awareness of MMT model and achieve superior results against strong baselines. 1 Introduction Multimodal machine translation (MMT) typically improves text translation quality by introducing an extra visual modality input. This task hypothesizes that the corresponding visual context is helpful for disambiguation and omission completion when the source sentence is ambiguous or even incorrect. Compared to the purely text-based machine trans- lation, the critical point of MMT is to find out an effective approach to integrating images into a MT model that makes the most of visual information. Most of the existing studies have dedicated their efforts to the way of extracting multi-granularity vi- sual features for integration (Calixto and Liu, 2017; Delbrouck and Dupont, 2017; Ive et al., 2019; Zhao et al., 2022; Li et al., 2022; Fang and Feng, 2022) or designing model architectures for better message passing across various modalities (Calixto et al., 2017, 2019; Yao and Wan, 2020; Yin et al., 2020; ∗Corresponding author Lin et al., 2020; Liu et al., 2021), which achieve promising performances in most of the multimodal scenarios. Despite the success of these methods, some re- searchers reveal that visual modality is underuti- lized by the modern MMT models (Grönroos et al., 2018; Elliott, 2018). Elliott (2018) leverage an ad- versarial evaluation method to appraise the aware- ness of visual contexts of MMT models. An in- teresting observation is that the disturbed image input generally shows no significant effects on the performance of public MMT models, indicating the insensitivity to the visual contexts. Recently, Wu et al. (2021) revisit the need for visual context in MMT and find that their strong MMT model tends to ignore the multimodal information. They suggest that the gains from the multimodal signals over text-only counterparts are, in fact, due to the regularization effect. In practice, the existing approaches mainly fo- cus on a better interaction method …

Content truncated to 500 words