CUNI System for the WMT17 Multimodal Translation Task

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 450–457 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics CUNI System for the WMT17 Multimodal Traslation Task Jindˇrich Helcl and Jindˇrich Libovick´y Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostransk´e n´amˇest´ı 25, 118 00 Prague, Czech Republic {helcl, libovicky}@ufal.mff.cuni.cz Abstract In this paper, we describe our submissions to the WMT17 Multimodal Translation Task. For Task 1 (multimodal translation), our best scoring system is a purely textual neural translation of the source image cap- tion to the target language. The main fea- ture of the system is the use of additional data that was acquired by selecting simi- lar sentences from parallel corpora and by data synthesis with back-translation. For Task 2 (cross-lingual image captioning), our best submitted system generates an English caption which is then translated by the best system used in Task 1. We also present negative results, which are based on ideas that we believe have potential of making improvements, but did not prove to be useful in our particular setup. 1 Introduction Recent advances in deep learning allowed infer- ring distributed vector representations of both tex- tual and visual data. In models combining text and vision modalities, this representation can be used as a shared data type. Unlike the classical nat- ural language processing tasks where everything happens within one language or across languages, multimodality tackles how the language entities relate to the extra-lingual reality. One of these tasks is multimodal translation whose goal is us- ing cross-lingual information in automatic image captioning. In this system-description paper, we describe our submission to the WMT17 Multimodal Trans- lation Task. In particular, we discuss the effect of mining additional training data and usability of ad- vanced attention strategies. We report our results on both the 2016 and 2017 test sets and discuss efficiency of tested approaches. The rest of the paper is organized as follows. Section 2 introduces the tasks we handle in this pa- per and the datasets that were provided to the task. Section 3 summarizes the state-of-the-art methods applied to the task. In Section 4, we describe our models and the results we have achieved. Sec- tion 5 presents the negative results and Section 6 concludes the paper. 2 Task and Dataset Description The challenge of the WMT Multimodal Transla- tion Task is to exploit cross-lingual information in automatic image caption generation. The state- of-the-art models in both machine translation and automatic image caption generation use similar architectures for generating the target sentence. The simplicity with which we can combine the learned representations of various inputs in a sin- gle deep learning model inevitably leads to a ques- tion whether combining the modalities can lead to some interesting results. In the shared task, this is explored in two subtasks with different roles of visual and textual modalities. In the multimodal translation task (Task 1), the input of the model is an image and its …

Content truncated to 500 words