Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning

Proceedings of NAACL-HLT 2016, pages 171–181, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning Janarthanan Rajendran IIT Madras, India. rsdjjana@gmail.com Mitesh M. Khapra IBM Research India. mikhapra@in.ibm.com Sarath Chandar University of Montreal. apsarathchandar@gmail.com Balaraman Ravindran IIT Madras, India. ravi@cse.iitm.ac.in Abstract Recently there has been a lot of interest in learning common representations for multiple views of data. Typically, such common rep- resentations are learned using a parallel cor- pus between the two views (say, 1M images and their English captions). In this work, we address a real-world scenario where no direct parallel data is available between two views of interest (say, V1 and V2) but parallel data is available between each of these views and a pivot view (V3). We propose a model for learning a common representation for V1, V2 and V3 using only the parallel data avail- able between V1V3 and V2V3. The proposed model is generic and even works when there are n views of interest and only one pivot view which acts as a bridge between them. There are two specific downstream applications that we focus on (i) transfer learning between lan- guages L1,L2,...,Ln using a pivot language L and (ii) cross modal access between im- ages and a language L1 using a pivot lan- guage L2. Our model achieves state-of-the-art performance in multilingual document classi- fication on the publicly available multilingual TED corpus and promising results in multilin- gual multimodal retrieval on a new dataset cre- ated and released as a part of this work. 1 Introduction The proliferation of multilingual and multimodal content online has ensured that multiple views of the same data exist. For example, it is common to find the same article published in multiple lan- guages online in multilingual news articles, multilin- gual wikipedia articles, etc. Such multiple views can even belong to different modalities. For example, images and their textual descriptions are two views of the same entity. Similarly, audio, video and subti- tles of a movie are multiple views of the same entity. Learning common representations for such mul- tiple views of data will help in several downstream applications. For example, learning a common rep- resentation for images and their textual descriptions could help in finding images which match a given textual description. Further, such common represen- tations can also facilitate transfer learning between views. For example, a document classifier trained on one language (view) can be used to classify doc- uments in another language by representing docu- ments of both languages in a common subspace. Existing approaches to common representation learning (Ngiam et al., 2011; Klementiev et al., 2012; Chandar et al., 2013; Chandar et al., 2014; Andrew et al., 2013; Wang et al., 2015) except (Her- mann and Blunsom, 2014b) typically require paral- lel data between all views. However, in many real- world scenarios such parallel data may not be avail- able. For example, while there are many publicly available datasets containing images and their …

Content truncated to 500 words