A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3025–3035 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3025 A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation Yongjing Yin1∗, Fandong Meng2, Jinsong Su1†, Chulun Zhou1, Zhengyuan Yang3, Jie Zhou2, Jiebo Luo3 1Xiamen University, Xiamen, China 2Pattern Recognition Center, WeChat AI, Tencent Inc, Beijing, China 3Department of Computer Science, University of Rochester, Rochester NY 14627, USA yinyongjing@stu.xmu.edu.cn fandongmeng@tencent.com jssu@xmu.edu.cn Abstract Multi-modal neural machine translation (NMT) aims to translate source sentences into a target language paired with images. However, dominant multi-modal NMT models do not fully exploit fine-grained semantic correspondences between semantic units of different modalities, which have potential to refine multi-modal representation learning. To deal with this issue, in this paper, we propose a novel graph-based multi-modal fusion en- coder for NMT. Specifically, we first represent the input sentence and image using a unified multi-modal graph, which captures various semantic relationships between multi-modal semantic units (words and visual objects). We then stack multiple graph-based multi-modal fusion layers that iteratively perform semantic interactions to learn node representations. Finally, these representations provide an attention-based context vector for the decoder. We evaluate our proposed encoder on the Multi30K datasets. Experimental results and in-depth analysis show the superiority of our multi-modal NMT model. 1 Introduction Multi-modal neural machine translation (NMT) (Huang et al., 2016; Calixto et al., 2017) has be- come an important research direction in machine translation, due to its research significance in multi- modal deep learning and wide applications, such as translating multimedia news and web product infor- mation (Zhou et al., 2018). It significantly extends the conventional text-based machine translation by taking images as additional inputs. The assump- tion behind this is that the translation is expected to be more accurate compared to purely text-based ∗This work is done when Yongjing Yin was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. †Corresponding author. translation, since the visual context helps to resolve ambiguous multi-sense words (Ive et al., 2019). Apparently, how to fully exploit visual informa- tion is one of the core issues in multi-modal NMT, which directly impacts the model performance. To this end, a lot of efforts have been made, roughly consisting of: (1) encoding each input image into a global feature vector, which can be used to ini- tialize different components of multi-modal NMT models, or as additional source tokens (Huang et al., 2016; Calixto et al., 2017), or to learn the joint multi-modal representation (Zhou et al., 2018; Cal- ixto et al., 2019); (2) extracting object-based im- age features to initialize the model, or supplement source sequences, or generate attention-based vi- sual context (Huang et al., 2016; Ive et al., 2019); and (3) representing each image as spatial features, which can be exploited as extra context (Calixto et al., 2017; Delbrouck and Dupont, 2017a; Ive et al., 2019), or a supplement to source semantics (Delbrouck and Dupont, 2017b) via an attention mechanism. Despite their success, the above studies do …

Content truncated to 500 words