Double Attention-based Multimodal Neural Machine Translation with Semantic Image Regions

Double Attention-based Multimodal Neural Machine Translation with Semantic Image Regions Yuting Zhao1, Mamoru Komachi1, Tomoyuki Kajiwara2, Chenhui Chu2 1Tokyo Metropolitan University, 6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan 2Osaka University, 2-8 Yamadaoka, Suita, Osaka 565-0871, Japan zhao-yuting@ed.tmu.ac.jp komachi@tmu.ac.jp {kajiwara,chu}@ids.osaka-u.ac.jp Abstract Existing studies on multimodal neural ma- chine translation (MNMT) have mainly focused on the effect of combining vi- sual and textual modalities to improve translations. However, it has been sug- gested that the visual modality is only marginally beneficial. Conventional vi- sual attention mechanisms have been used to select the visual features from equally- sized grids generated by convolutional neural networks (CNNs), and may have had modest effects on aligning the vi- sual concepts associated with textual ob- jects, because the grid visual features do not capture semantic information. In con- trast, we propose the application of se- mantic image regions for MNMT by in- tegrating visual and textual features us- ing two individual attention mechanisms (double attention). We conducted ex- periments on the Multi30k dataset and achieved an improvement of 0.5 and 0.9 BLEU points for English→German and English→French translation tasks, com- pared with the MNMT with grid visual features. We also demonstrated concrete improvements on translation performance benefited from semantic image regions. 1 Introduction Neural machine translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015) has achieved state-of-the-art translation performance. Recently, c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. Object detection man in red shirt watches dog on an agility course. RNN attention attention un homme en polo rouge regarde son chien sur un parcours d's agilité . Image regions Figure 1: Overview of our MNMT model. many studies (Specia et al., 2016; Elliott et al., 2017; Barrault et al., 2018) have been increas- ingly focusing on incorporating multimodal con- tents, particularly images, to improve translations. Hence, researchers in this field have established a shared task called multimodal machine translation (MMT), which consists of translating a target sen- tence from a source language description into an- other language using information from the image described by the source sentence. The first MMT study by (Elliott et al., 2015) demonstrated the potential of improving the trans- lation quality by using image. To effectively use an image, several subsequent studies (Gao et al., 2015; Huang et al., 2016; Calixto and Liu, 2017) incorporated global visual features extracted from the entire image by convolutional neural networks (CNNs) into a source word sequence or hidden states of a recurrent neural network (RNN). Fur- thermore, other studies started using local visual features in the context of an attention-based NMT. These features were extracted from equally-sized grids in an image by a CNN. For instance, multi- modal attention (Caglayan et al., 2016b) has been designed for a mix of text and local visual fea- tures. Additionally, double attention mechanisms (Calixto et al., 2017) have been proposed for text st-1 st homme en chemise rouge <eos> yt-1 yt zt ct Decoder rp Faster R-CNN …

Content truncated to 500 words