DCU System Report on the WMT 2017 Multi-modal Machine Translation Task

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 440–444 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics DCU System Report on the WMT 2017 Multi-modal Machine Translation Task Iacer Calixto and Koel Dutta Chowdhury and Qun Liu {iacer.calixto,koel.chowdhury,qun.liu}@adaptcentre.ie Abstract We report experiments with multi-modal neural machine translation models that in- corporate global visual features in differ- ent parts of the encoder and decoder, and use the VGG19 network to extract fea- tures for all images. In our experiments, we explore both different strategies to in- clude global image features and also how ensembling different models at inference time impact translations. Our submissions ranked 3rd best for translating from En- glish into French, always improving con- siderably over an neural machine transla- tion baseline across all language pair eval- uated, e.g. an increase of 7.0–9.2 ME- TEOR points. 1 Introduction In this paper we report on our application of three different multi-modal neural machine translation (NMT) systems to translate image descriptions. We use encoder–decoder attentive multi-modal NMT models where each training example con- sists of one source variable-length sequence, one image, and one target variable-length sequence, and a model is trained to translate sequences in the source language into corresponding sequences in the target language while taking the image into consideration. We use the three models introduced in Calixto et al. (2017b), which integrate global image features extracted using a pre-trained con- volutional neural network into NMT (i) as words in the source sentence, (ii) to initialise the encoder hidden state, and (iii) as additional data to initialise the decoder hidden state. We are inspired by the recent success of multi- modal NMT models applied to the translation of image descriptions (Huang et al., 2016; Calixto et al., 2017a). Huang et al. (2016) incorporate global visual features into NMT with some suc- cess, and Calixto et al. (2017a) propose to use lo- cal visual features instead, achieving better results. We follow Calixto et al. (2017b) and investigate whether we can achieve better results while still using global visual features, which are consider- ably smaller and simpler to integrate when com- pared to local features. We expect that, by integrating visual informa- tion when translating image descriptions, we are able to exploit valuable information from both modalities when generating the target description, effectively grounding machine translation (Glen- berg and Robertson, 2000). 2 Model Description The models used in our experiments can be viewed as expansions of the attentive NMT framework in- troduced by Bahdanau et al. (2015) with the addi- tion of a visual component that incorporates visual features from images. A bi-directional recurrent neural network (RNN) with gated recurrent unit (GRU) (Cho et al., 2014) is used as the encoder. The final annotation vector for a given source po- sition i is the concatenation of forward and back- ward RNN hidden states, hi = �−→ hi; ←− hi .� We use the publicly available pre-trained con- volution neural network VGG191 of Simonyan and …

Content truncated to 500 words