Supervised Visual Attention for Multimodal Neural Machine Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4304–4314 Barcelona, Spain (Online), December 8-13, 2020 4304 Supervised Visual Attention for Multimodal Neural Machine Translation Tetsuro Nishihara Ehime University t nishihara@ai.cs.ehime-u.ac.jp Akihiro Tamura Doshisha University aktamura@mail.doshisha.ac.jp Takashi Ninomiya Ehime University ninomiya@cs.ehime-u.ac.jp Yutaro Omote Ehime University omote@ai.cs.ehime-u.ac.jp Hideki Nakayama The University of Tokyo nakayama@ci.i.u-tokyo.ac.jp Abstract This paper proposed a supervised visual attention mechanism for multimodal neural machine translation (MNMT), trained with constraints based on manual alignments between words in a sentence and their corresponding regions of an image. The proposed visual attention mechanism captures the relationship between a word and an image region more precisely than a conven- tional visual attention mechanism trained through MNMT in an unsupervised manner. Our ex- periments on English-German and German-English translation tasks using the Multi30k dataset and on English-Japanese and Japanese-English translation tasks using the Flickr30k Entities JP dataset show that a Transformer-based MNMT model can be improved by incorporating our proposed supervised visual attention mechanism and that further improvements can be achieved by combining it with a supervised cross-lingual attention mechanism (up to +1.61 BLEU, +1.7 METEOR). 1 Introduction As mainstream machine translation, Neural Machine Translation (NMT) model, widely used since the early days, is the Recurrent Neural Network (RNN)-based NMT with attention mechanism (Luong et al., 2015). This model achieves higher translation accuracy than conventional RNN-based NMT by using a cross-lingual attention mechanism that captures the relationship between words in source and target language sentences. In recent years, the Transformer model (Vaswani et al., 2017) has been attracting much attention because it achieves higher accuracy than methods using RNN and Convolutional Neural Network (CNN). In addition to the conventional cross-lingual attention mechanism, the Transformer model introduces a self-attention mechanism that captures relationships between words in a sentence. Various studies have been conducted on methods to improve NMT’s performance and on one method constraining the cross-lingual attention mechanism (Liu et al., 2016; Mi et al., 2016; Garg et al., 2019). These cited researchers have improved translation performance by using a tool to obtain alignments between words in source and target language sentences in advance and then providing alignments as supervisions to train the cross-lingual attention mechanism. Multimodal NMT (MNMT) (Barrault et al., 2018) is a machine translation task that aims to improve translation performance by using images in addition to source language sentences because input images are considered useful for disambiguation and omission completion. Helcl et al. (2018) have proposed an MNMT model that introduces a visual attention mechanism into the Transformer-based MNMT model’s decoder to capture alignment between words in a sentence and image regions to utilize image features from CNNs. Additionally, Delbrouck and Dupont (2017) have proposed a model that introduces a vi- sual attention mechanism into an RNN-based MNMT model’s encoder. However, these visual attention mechanisms are automatically trained through MNMT in an unsupervised manner, and, hence, they do not always capture the relationship between words in a sentence and image regions that they should. This paper proposes a supervised …

Content truncated to 500 words