Incorporating Global Visual Features into Attention-based Neural Machine Translation.

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 992–1003 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Incorporating Global Visual Features into Attention-Based Neural Machine Translation Iacer Calixto ADAPT Centre Dublin City University School of Computing Glasnevin, Dublin 9 iacer.calixto@adaptcentre.ie Qun Liu ADAPT Centre Dublin City University School of Computing Glasnevin, Dublin 9 qun.liu@adaptcentre.ie Abstract We introduce multi-modal, attention- based Neural Machine Translation (NMT) models which incorporate visual features into different parts of both the encoder and the decoder. Global image features are extracted using a pre-trained convo- lutional neural network and are incorpo- rated (i) as words in the source sentence, (ii) to initialise the encoder hidden state, and (iii) as additional data to initialise the decoder hidden state. In our exper- iments, we evaluate translations into En- glish and German, how different strategies to incorporate global image features com- pare and which ones perform best. We also study the impact that adding synthetic multi-modal, multilingual data brings and find that the additional data have a posi- tive impact on multi-modal models. We report new state-of-the-art results and our best models also significantly improve on a comparable Phrase-Based Statistical MT (PBSMT) model trained on the Multi30k data set according to all metrics evaluated. To the best of our knowledge, it is the first time a purely neural model significantly improves over a PBSMT model on all met- rics evaluated on this data set. 1 Introduction Neural Machine Translation (NMT) has recently been proposed as an instantiation of the sequence to sequence (seq2seq) learning problem (Kalch- brenner and Blunsom, 2013; Cho et al., 2014b; Sutskever et al., 2014). In this problem, each train- ing example consists of one source and one tar- get variable-length sequence, with no prior infor- mation regarding the alignments between the two. A model is trained to translate sequences in the source language into corresponding sequences in the target. This framework has been successfully used in many different tasks, such as handwritten text generation (Graves, 2013), image description generation (Hodosh et al., 2013; Kiros et al., 2014; Mao et al., 2014; Elliott et al., 2015; Karpathy and Fei-Fei, 2015; Vinyals et al., 2015), machine trans- lation (Cho et al., 2014b; Sutskever et al., 2014) and video description generation (Donahue et al., 2015; Venugopalan et al., 2015). Recently, there has been an increase in the num- ber of natural language generation models that explicitly use attention-based decoders, i.e. de- coders that model an intra-sequential mapping be- tween source and target representations. For in- stance, Xu et al. (2015) proposed an attention- based model for the task of Image Description Generation (IDG) where the model learns to at- tend to specific parts of an image (the source) as it generates its description (the target). In MT, one can intuitively interpret this attention mechanism as inducing an alignment between source and tar- get sentences, as first proposed by Bahdanau et al. (2015). The common idea is to explicitly frame a learning task …

Content truncated to 500 words