Multimodal Pivots for Image Caption Translation

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 2399–2409, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Multimodal Pivots for Image Caption Translation Julian Hitschler and Shigehiko Schamoni Computational Linguistics Heidelberg University 69120 Heidelberg, Germany {hitschler,schamoni}@cl.uni-heidelberg.de Stefan Riezler Computational Linguistics & IWR Heidelberg University 69120 Heidelberg, Germany riezler@cl.uni-heidelberg.de Abstract We present an approach to improve sta- tistical machine translation of image de- scriptions by multimodal pivots defined in visual space. The key idea is to perform image retrieval over a database of images that are captioned in the target language, and use the captions of the most similar images for crosslingual reranking of trans- lation outputs. Our approach does not de- pend on the availability of large amounts of in-domain parallel data, but only re- lies on available large datasets of monolin- gually captioned images, and on state-of- the-art convolutional neural networks to compute image similarities. Our experi- mental evaluation shows improvements of 1 BLEU point over strong baselines. 1 Introduction Multimodal data consisting of images and natural language descriptions (henceforth called captions) are an abundant source of information that has led to a recent surge in research integrating language and vision. Recently, the aspect of multilinguality has been added to multimodal language process- ing in a shared task at the WMT16 conference.1 There is clearly also a practical demand for mul- tilingual image captions, e.g., automatic transla- tion of descriptions of art works would allow ac- cess to digitized art catalogues across language barriers and is thus of social and cultural interest; multilingual product descriptions are of high com- mercial interest since they would allow to widen e-commerce transactions automatically to interna- tional markets. However, while datasets of images and monolingual captions already include millions 1http://www.statmt.org/wmt16/ multimodal-task.html of tuples (Ferraro et al., 2015), the largest multi- lingual datasets of images and captions known to the authors contain 20,000 (Grubinger et al., 2006) or 30,0002 triples of images with German and En- glish descriptions. In this paper, we want to address the problem of multilingual captioning from the perspective of statistical machine translation (SMT). In contrast to prior work on generating captions directly from images (Kulkarni et al. (2011), Karpathy and Fei- Fei (2015), Vinyals et al. (2015), inter alia), our goal is to integrate visual information into an SMT pipeline. Visual context provides orthogonal in- formation that is free of the ambiguities of natu- ral language, therefore it serves to disambiguate and to guide the translation process by ground- ing the translation of a source caption in the ac- companying image. Since datasets consisting of source language captions, images, and target lan- guage captions are not available in large quanti- ties, we would instead like to utilize large datasets of images and target-side monolingual captions to improve SMT models trained on modest amounts of parallel captions. Let the task of caption translation be defined as follows: For production of a target caption ei of an image i, a system may use as …

Content truncated to 500 words