Noise-robust Cross-modal Interactive Learning with Text2Image Mask for Multi-modal Neural Machine Translation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5098–5108 October 12–17, 2022. 5098 Noise-robust Cross-modal Interactive Learning with Text2image Mask for Multi-modal Neural Machine Translation Junjie Ye1,2, Junjun Guo1,2∗, Yan Xiang1,2, Kaiwen Tan1,2, Zhengtao Yu1,2 1 Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China 2 Yunnan Key Laboratory of Artificial Intelligence, Kunming, China junjieye.cdx@qq.com, guojjgb@163.com, sharonxiang@126.com, kwtan0909@qq.com, ztyu@hotmail.com Abstract Multi-modal neural machine translation (MNMT) aims to improve textual level machine translation performance in the presence of text-related images. Most of the previous works on MNMT focus on multi-modal fusion methods with full visual features. However, text and its corresponding image may not match exactly, visual noise is generally inevitable. The irrelevant image regions may mislead or distract the textual attention and cause model performance degra- dation. This paper proposes a noise-robust multi-modal interactive fusion approach with cross-modal relation-aware mask mechanism for MNMT. A text-image relation-aware attention module is constructed through the cross-modal interaction mask mechanism, and visual features are extracted based on the text-image interaction mask knowledge. Then a noise-robust multi-modal adaptive fusion approach is presented by fusion the relevant visual and textual features for machine translation. We validate our method on the Multi30K dataset. The experimental results show the superiority of our proposed model, and achieve the state-of-the-art scores in all En-De, En-Fr and En-Cs translation tasks 1. 1 Introduction Multi-modal Neural Machine Translation (MNMT) aims to optimize the conventional text-only ma- chine translation systems by using multi-modal information (eg., image, video, sound), which has received growing research attentions in the fields of CV and NLP, recently. A reasonable assumption is that visual information is helpful to improve textual-level machine translation (El- liott et al., 2017; Barrault et al., 2018; Ye and Guo, 2022), and many studies have been carried out to conduct the benefits of image for NMT (Caglayan ∗Corresponding author. 1https://github.com/nlp-mmt/Noise-robust-Text2image- Mask et al., 2019; Yin et al., 2020; Li et al., 2021a). As expected, the fusion of visual information actually improves the performance of machine translation (Caglayan et al., 2019). Most existing MNMT methods mainly focus on how to design a excellent multi-modal fusion framework to bridge the semantic gap between im- age and text, while visual noise is often ignored. Unfortunately, it is often difficult to obtain the image that exactly match the textual information. What is worse, image information and textual infor- mation may even be weakly correlated with each other. Visual noise is generally unavoidable(Li et al., 2022; Yao and Wan, 2020). As shown in Figure 1 (left), objects such as old man, brown hat and bench included in the image, these visual objects correspond to the ’old man’, ’brown hat’ and ’bench’ in the source sentence, which is the useful visual information for machine translation. However, the image also contains some irrelevant visual information ( e.g., tree, flower, grasse) for the source sentence, the mismatched visual-textual information may distract the multi-modal fusion and then lead to machine translation performance decay. Therefore, it is necessary …

Content truncated to 500 words