Image-Mediated Learning for Zero-Shot Cross-Lingual Document Retrieval

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 585–590, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Image-Mediated Learning for Zero-Shot Cross-Lingual Document Retrieval Ruka Funaki, Hideki Nakayama Machine Perception Group Graduate School of Information Science and Technology The University of Tokyo {funaki,nakayama}@nlab.ci.i.u-tokyo.ac.jp Abstract We propose an image-mediated learning approach for cross-lingual document re- trieval where no or only a few parallel corpora are available. Using the images in image-text documents of each language as the hub, we derive a common seman- tic subspace bridging two languages by means of generalized canonical correla- tion analysis. For the purpose of evalu- ation, we create and release a new doc- ument dataset consisting of three types of data (English text, Japanese text, and images). Our approach substantially en- hances retrieval accuracy in zero-shot and few-shot scenarios where text-to-text ex- amples are scarce. 1 Introduction Cross-lingual document retrieval (CLDR) is the task of finding relevant documents in one lan- guage given a query document in another lan- guage. While sufficiently large-scale corpora are critical for parallel corpus-based learning meth- ods, manually creating corpora requires huge hu- man effort and is unrealistic in many cases. A straightforward approach is to crawl bilin- gual documents from the Web for use as train- ing data. However, because most documents on the Web are written in one language, it is not al- ways easy to collect a sufficient number of multi- lingual documents, especially those involving mi- nor languages. Let us consider the multimedia information in documents. We can, for exam- ple, find abundant pairings of text and images, e.g., text with the ALT property of <IMG> tags in HTML, text with photos posted to social network- ing sites, and articles on Web news posted with images. Unlike text, an image is a universal rep- resentation; we can easily understand the semantic Learn relation indirectly Language 1 Documents Language 2 Documents images images pairs pairs similarity Image-Mediated Learning Language1 Texts Language2 Texts Figure 1: Concept of image-mediated learning. Our idea is to learn the relation between two lan- guages indirectly by using images attached to text. If two documents written in different languages include images with similar image features, it is likely that the texts contained in the two docu- ments are similar. Based on this idea, we seek the relation of texts written in different languages me- diated by the similarity between images. content of images regardless of our mother tongue. Motivated by this observation, we expect that we can learn the relation of two languages indirectly through images, even if we do not have sufficient bilingual text pairs (Figure 1). Generally, traditional image recognition tech- niques (or image features) are very poor com- pared with those in the natural language process- ing field. In recent years, however, deep learning has resulted in a breakthrough in visual recogni- tion and dramatically improved image recognition accuracy in generic domains, which is rapidly ap- proaching human recognition levels (Fang et al., …

Content truncated to 500 words