Image Pivoting for Learning Multilingual Multimodal Representations

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2839–2845 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Image Pivoting for Learning Multilingual Multimodal Representations Spandana Gella Rico Sennrich Frank Keller Mirella Lapata Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh {spandana.gella, rico.sennrich}@ed.ac.uk {keller,mlap}@inf.ed.ac.uk Abstract In this paper we propose a model to learn multimodal multilingual representa- tions for matching images and sentences in different languages, with the aim of advancing multilingual versions of image search and image understanding. Our model learns a common representation for images and their descriptions in two dif- ferent languages (which need not be paral- lel) by considering the image as a pivot be- tween two languages. We introduce a new pairwise ranking loss function which can handle both symmetric and asymmetric similarity between the two modalities. We evaluate our models on image-description ranking for German and English, and on semantic textual similarity of image de- scriptions in English. In both cases we achieve state-of-the-art performance. 1 Introduction In recent years there has been a significant amount of research in language and vision tasks which require the joint modeling of texts and images. Examples include text-based image retrieval, im- age description and visual question answering. An increasing number of large image descrip- tion datasets has become available (Hodosh et al., 2013; Young et al., 2014; Lin et al., 2014) and various systems have been proposed to handle the image description task as a generation problem (Bernardi et al., 2016; Mao et al., 2015; Vinyals et al., 2015; Fang et al., 2015). There has also been a great deal of work on sentence-based im- age search or cross-modal retrieval where the ob- jective is to learn a joint space for images and text (Hodosh et al., 2013; Frome et al., 2013; Karpathy et al., 2014; Kiros et al., 2015; Socher et al., 2014; Donahue et al., 2015). Previous work on image description generation or learning a joint space for images and text has mostly focused on English due to the availabil- ity of English datasets. Recently there have been attempts to create image descriptions and mod- els for other languages (Funaki and Nakayama, 2015; Elliott et al., 2016; Rajendran et al., 2016; Miyazaki and Shimizu, 2016; Specia et al., 2016; Li et al., 2016; Hitschler et al., 2016; Yoshikawa et al., 2017). Most work on learning a joint space for images and their descriptions is based on Canonical Cor- relation Analysis (CCA) or neural variants of CCA over representations of image and its descriptions (Hodosh et al., 2013; Andrew et al., 2013; Yan and Mikolajczyk, 2015; Gong et al., 2014; Chan- dar et al., 2016). Besides CCA, a few others learn a visual-semantic or multimodal embedding space of image descriptions and representations by opti- mizing a ranking cost function (Kiros et al., 2015; Socher et al., 2014; Ma et al., 2015; Vendrov et al., 2016) or by aligning image regions (objects) and segments of the description …

Content truncated to 500 words