An empirical study on the effectiveness of images in Multimodal Neural Machine Translation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 910–919 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics An empirical study on the effectiveness of images in Multimodal Neural Machine Translation Jean-Benoit Delbrouck and St´ephane Dupont TCTS Lab, University of Mons, Belgium {jean-benoit.delbrouck, stephane.dupont}@umons.ac.be Abstract In state-of-the-art Neural Machine Trans- lation (NMT), an attention mechanism is used during decoding to enhance the trans- lation. At every step, the decoder uses this mechanism to focus on different parts of the source sentence to gather the most use- ful information before outputting its tar- get word. Recently, the effectiveness of the attention mechanism has also been ex- plored for multimodal tasks, where it be- comes possible to focus both on sentence parts and image regions that they describe. In this paper, we compare several atten- tion mechanism on the multimodal trans- lation task (English, image → German) and evaluate the ability of the model to make use of images to improve translation. We surpass state-of-the-art scores on the Multi30k data set, we nevertheless iden- tify and report different misbehavior of the machine while translating. 1 Introduction In machine translation, neural networks have at- tracted a lot of research attention. Recently, the attention-based encoder-decoder framework (Sutskever et al., 2014; Bahdanau et al., 2014) has been largely adopted. In this approach, Recurrent Neural Networks (RNNs) map source sequences of words to target sequences. The attention mech- anism is learned to focus on different parts of the input sentence while decoding. Attention mecha- nisms have shown to work with other modalities too, like images, where their are able to learn to attend the salient parts of an image, for instance when generating text captions (Xu et al., 2015). For such applications, Convolutional Neural Net- works (CNNs) such as Deep Residual (He et al., 2016) have shown to work best to represent im- ages. Multimodal models of texts and images em- power new applications such as visual question an- swering or multimodal caption translation. Also, the grounding of multiple modalities against each other may enable the model to have a better under- standing of each modality individually, such as in natural language understanding applications. In the field of Machine Translation (MT), the ef- ficient integration of multimodal information still remains a challenging task. It requires combining diverse modality vector representations with each other. These vector representations, also called context vectors, are computed in order the capture the most relevant information in a modality to out- put the best translation of a sentence. To investigate the effectiveness of informa- tion obtained from images, a multimodal machine translation shared task (Specia et al., 2016) has been addressed to the MT community1. The best results of NMT model were those of Huang et al. (2016) who used LSTM fed with global visual features or multiple regional visual features fol- lowed by rescoring. Recently, Calixto et al. (2017) proposed a doubly-attentive decoder that outper- formed this baseline with less data and without rescoring. Our …

Content truncated to 500 words