NICT-NAIST System for WMT17 Multimodal Translation Task

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 477–482 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics NICT-NAIST System for WMT17 Multimodal Translation Task Jingyi Zhang1,2, Masao Utiyama1, Eiichro Sumita1 Graham Neubig2, Satoshi Nakamura2 1National Institute of Information and Communications Technology, 3-5Hikaridai, Keihanna Science City, Kyoto 619-0289, Japan 2Graduate School of Information Science, Nara Institute of Science and Technology, Takayama, Ikoma, Nara 630-0192, Japan jingyizhang/mutiyama/eiichiro.sumita@nict.go.jp neubig/s-nakamura@is.naist.jp Abstract This paper describes the NICT-NAIST system for the WMT 2017 shared multi- modal machine translation task for both language pairs, English-to-German and English-to-French. We built a hierarchical phrase-based (Hiero) translation system and trained an attentional encoder-decoder neural machine translation (NMT) model to rerank the n-best output of the Hiero system, which obtained significant gains over both the Hiero system and NMT de- coding alone. We also present a multi- modal NMT model that integrates the tar- get language descriptions of images that are similar to the image described by the source sentence as additional inputs of the neural networks to help the translation of the source sentence. We give detailed analysis for the results of the multimodal NMT model. Our system obtained the first place for the English-to-French task ac- cording to human evaluation. 1 Introduction We participated in the WMT 2017 shared mul- timodal machine translation task 1, which trans- lates a source language description of an image into a target language description. We built sys- tems for both English-to-German and English-to- French language pairs. Our baseline systems only use text information. We compared three text-only approaches: a hier- archical phrase-based (Hiero) translation system (Chiang, 2005), an attentional encoder-decoder neural machine translation (NMT) system (Bah- danau et al., 2015), and a system using the NMT model to rerank the n-best output of the Hiero sys- tem. We also explored ways to improve the NMT model with image information. Compared to pre- vious multimodal NMT (MNMT) models that in- tegrate visual features directly (Caglayan et al., 2016; Calixto et al., 2016; Huang et al., 2016; Cal- ixto et al., 2017), we first exploit image retrieval methods to obtain images that are similar to the image described by the source sentence, and then integrate the target language descriptions of these similar images into the NMT model to help the translation of the source sentence. This makes it possible to exploit a large corpus with only images and target language descriptions through an image retrieval step. This is similar to Hitschler et al. (2016)’s multimodal pivots method, which uses target descriptions of similar images for reranking MT outputs, while we use these target descriptions as additional inputs for the NMT model. 2 Text-only MT We compared three text-only approaches for this translation task. 2.1 Hierarchical Phrase-based SMT The hierarchical phrase-based model (Chiang, 2005) extracts hierarchical phrase-based transla- tion rules from parallel sentence pairs with word alignments. The word alignments can be learned by IBM models. Each translation rule contains several feature scores. The decoder of hierarchi- cal …

Content truncated to 500 words