Doubly-Attentive Decoder for Multi-modal Neural Machine Translation

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1913–1924 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1175 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1913–1924 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1175 Doubly-Attentive Decoder for Multi-modal Neural Machine Translation Iacer Calixto ADAPT Centre School of Computing Dublin City University Dublin, Ireland Qun Liu ADAPT Centre School of Computing Dublin City University Dublin, Ireland {iacer.calixto,qun.liu,nick.campbell}@adaptcentre.ie Nick Campbell ADAPT Centre Speech Communication Lab Trinity College Dublin Dublin 2, Ireland Abstract We introduce a Multi-modal Neural Ma- chine Translation model in which doubly-attentive decoder naturally incor-a porates spatial visual features obtained us- ing pre-trained convolutional neural net- works, bridging the gap between image description and translation. Our decoder learns to attend to source-language words and parts of an image independently by means of two separate attention mecha- nisms as it generates words in the target language. We find that our model can efficiently exploit not just back-translated in-domain multi-modal data but also large general-domain text-only MT corpora. We also report state-of-the-art results on the Multi30k data set. 1 Introduction Neural Machine Translation (NMT) has been suc- cessfully tackled as a sequence to sequence learn- ing problem (Kalchbrenner and Blunsom, 2013; Cho et al., 2014b; Sutskever et al., 2014) where each training example consists of one source and one target variable-length sequences, with no prior information on the alignment between the two. In the context of NMT, Bahdanau et al. (2015) first proposed to use an attention mechanism in the decoder, which is trained to attend to the rel- evant source-language words as it generates each word of the target sentence. Similarly, Xu et al. (2015) proposed an attention-based model for the task of image description generation (IDG) where a model learns to attend to specific parts of an im- age representation (the source) as it generates its description (the target) in natural language. We are inspired by recent successes in applying attention-based models to NMT and IDG. In this work, we propose an end-to-end attention-based multi-modal neural machine translation (MNMT) model which effectively incorporates two inde- pendent attention mechanisms, one over source- language words and the other over different areas of an image. Our main contributions are: • We propose a novel attention-based MNMT model which incorporates spatial visual fea- tures in a separate visual attention mecha- nism; • We use a medium-sized, back-translated multi-modal in-domain data set and large general-domain text-only MT corpora to pre- train our models and show that our MNMT model can efficiently exploit both; • We show that images bring useful informa- tion into an NMT model, e.g. in situations in which sentences describe objects illustrated in the image. To the best of our knowledge, previous MNMT models in the literature that utilised spatial vi- sual features did not significantly improve over a comparable model that used global visual fea- tures or even only …

Content truncated to 500 words