Generating Image Descriptions using Multilingual Data

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 458–464 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Generating Image Descriptions using Multilingual Data Alan Jaffe School of Computer Science Carnegie Mellon University Pittsburgh, PA apjaffe@andrew.cmu.edu Abstract In this paper we explore several neural network architectures for the WMT 2017 multimodal translation sub-task on mul- tilingual image caption generation. The goal of the task is to generate image cap- tions in German, using a training corpus of images with captions in both English and German. We explore several mod- els which attempt to generate captions for both languages, ignoring the English out- put during evaluation. We compare the re- sults to a baseline implementation which uses only the German captions for training and show significant improvement. 1 Introduction Neural models have shown great success on a variety of tasks, including machine transla- tion (Sutskever et al., 2014), image caption gen- eration (Xu et al., 2015), and language modeling (Bengio et al., 2003). Recently, huge datasets nec- essary for training these models have become more widely available, but there are still many lim- itations. In some cases, the dataset which is avail- able may not match the domain of the task. In this paper, we attempt to generate image cap- tions in German, using a training corpus of images with captions in both English and German. For each image, we have 5 independently generated captions in each language. Since the training cor- pus is relatively small (less than 30,000 images), we want to make use of the English language data to improve the German captions. (See figure 1). It is important to note that since these captions were generated independently in each language rather than translated, they often differ from each other quite a bit. Not only do they often choose to describe different features of an image, but also they sometimes describe contradictory features of the image (one caption describing a man sleeping on a couch while a different caption describes a woman sleeping on a couch). This inconsistency and the relatively small amount of training data makes it very difficult to train a reliable transla- tion system between the languages based on this corpus. In this paper, we will start by discussing related work in image caption generation. Then we will explain the baseline German image caption gen- eration model, the soft attention model from Xu et al. (2015). Several methods of incorporating the English data to improve the performance will be described. Finally, the experimental setup will be specified and the results will be evaluated. 2 Related Work The task of multilingual image caption genera- tion has been previously explored by Elliott et al. (2015). Elliott et al. (2015) used an LSTM to gen- erate captions, using features from both a source- language multimodal model and a target-language multimodal model. Other previous work on mul- tilingual images such as Hitschler and Riezler (2016) has focused on image caption translation, where captions …

Content truncated to 500 words