Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 215–233 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description Desmond Elliott School of Informatics University of Edinburgh d.elliott@ed.ac.uk Stella Frank Centre for Language Evolution University of Edinburgh stella.frank@ed.ac.uk Lo¨ıc Barrault and Fethi Bougares LIUM University of Le Mans first.last@univ-lemans.fr Lucia Specia Department of Computer Science University of Sheffield l.specia@sheffield.ac.uk Abstract We present the results from the second shared task on multimodal machine trans- lation and multilingual image description. Nine teams submitted 19 systems to two tasks. The multimodal translation task, in which the source sentence is supplemented by an image, was extended with a new lan- guage (French) and two new test sets. The multilingual image description task was changed such that at test time, only the image is given. Compared to last year, mul- timodal systems improved, but text-only systems remain competitive. 1 Introduction The Shared Task on Multimodal Translation and Multilingual Image Description tackles the prob- lem of generating descriptions of images for lan- guages other than English. The vast majority of image description research has focused on English- language description due to the abundance of crowdsourced resources (Bernardi et al., 2016). However, there has been a significant amount of recent work on creating multilingual image de- scription datasets in German (Elliott et al., 2016; Hitschler et al., 2016; Rajendran et al., 2016), Turk- ish (Unal et al., 2016), Chinese (Li et al., 2016), Japanese (Miyazaki and Shimizu, 2016; Yoshikawa et al., 2017), and Dutch (van Miltenburg et al., 2017). Progress on this problem will be useful for native-language image search, multilingual e- commerce, and audio-described video for visually impaired viewers. The first empirical results for multimodal trans- lation showed the potential for visual context to improve translation quality (Elliott et al., 2015; Hitschler et al., 2016). This was quickly followed by a wider range of work in the first shared task at WMT 2016 (Specia et al., 2016). The current shared task consists of two subtasks: • Task 1: Multimodal translation takes an im- age with a source language description that is then translated into a target language. The training data consists of parallel sentences with images. • Task 2: Multilingual image description takes an image and generates a description in the target language without additional source language information at test time. The train- ing data, however, consists of images with independent descriptions in both source and target languages. The translation task has been extended to include a new language, French. This extension means the Multi30K dataset (Elliott et al., 2016) is now triple aligned, with English descriptions translated into both German and French. The description generation task has substantially changed since last year. The main difference is that source language descriptions are no longer observed for test images. This mirrors the real- world scenario in which a target-language speaker wants a description of image that does …

Content truncated to 500 words