OSU Multimodal Machine Translation System Report

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 465–469 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics OSU Multimodal Machine Translation System Report Mingbo Ma, Dapeng Li, Kai Zhao† and Liang Huang Department of EECS Oregon State University Corvallis, OR 97331, USA {mam, lidap, zhaok, liang.huang}@oregonstate.edu Abstract This paper describes Oregon State Uni- versity’s submissions to the shared WMT’17 task “multimodal translation task I”. In this task, all the sentence pairs are image captions in different languages. The key difference be- tween this task and conventional ma- chine translation is that we have cor- responding images as additional infor- mation for each sentence pair. In this paper, we introduce a simple but effective system which takes an im- age shared between different languages, feeding it into the both encoding and decoding side. We report our sys- tem’s performance for English-French and English-German with Flickr30K (in-domain) and MSCOCO (out-of- domain) datasets. Our system achieves the best performance in TER for English-German for MSCOCO dataset. 1 Introduction Natural language generation (NLG) is one of the most important tasks in natural lan- guage processing (NLP). It can be applied to a lot of interesting applications such like ma- chine translation, image captioning, question answering. In recent years, Recurrent Neu- ral Networks (RNNs) based approaches have shown promising performance in generating more fluent and meaningful sentences com- pared with conventional models such as rule- based model (Mirkovic et al., 2011), corpus- based n-gram models (Wen et al., 2015) and trainable generators (Stent et al., 2004). † Current address: Google Inc., 111 8th Avenue, New York, New York, USA. More recently, attention-based encoder- decoder models (Bahdanau et al., 2014) have been proposed to provide the decoder more ac- curate alignments to generate more relevant words. The remarkable ability of attention mechanisms quickly update the state-of-the- art performance on variety of NLG tasks, such as machine translation (Luong et al., 2015), image captioning (Xu et al., 2015; Yang et al., 2016), and text summarization (Rush et al., 2015; Nallapati et al., 2016). However, for multimodal translation (Elliott et al., 2015), where we translate a caption from one language into another given a correspond- ing image, we need to design a new model since the decoder needs to consider both language and images at the same time. This paper describes our participation in the WMT 2017 multimodal task 1. Our model feeds the image information to both the en- coder and decoder, to ground their hidden rep- resentation within the same context of image during training. In this way, during testing time, the decoder would generate more rele- vant words given the context of both source sentence and image. 2 Model Description For the neural-based machine translation model, the encoder needs to map sequence of word embeddings from the source side into an- other representation of the entire sequence us- ing recurrent networks. Then, in the second stage, decoder generates one word at a time with considering global (sentence …

Content truncated to 500 words