Visual Bilingual Lexicon Induction with Transferred ConvNet Features

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 148–158, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Visual Bilingual Lexicon Induction with Transferred ConvNet Features Douwe Kiela Computer Laboratory University of Cambridge douwe.kiela@cl.cam.ac.uk Ivan Vuli´c Department of Computer Science KU Leuven ivan.vulic@cs.kuleuven.be Stephen Clark Computer Laboratory University of Cambridge stephen.clark@cl.cam.ac.uk Abstract This paper is concerned with the task of bilingual lexicon induction using image- based features. By applying features from a convolutional neural network (CNN), we obtain state-of-the-art performance on a standard dataset, obtaining a 79% relative improvement over previous work which uses bags of visual words based on SIFT features. The CNN image-based approach is also compared with state-of-the-art lin- guistic approaches to bilingual lexicon in- duction, even outperforming these for one of three language pairs on another stan- dard dataset. Furthermore, we shed new light on the type of visual similarity met- ric to use for genuine similarity versus re- latedness tasks, and experiment with using multiple layers from the same network in an attempt to improve performance. 1 Introduction Bilingual lexicon induction is the task of finding words that share a common meaning across differ- ent languages. It plays an important role in a va- riety of tasks in information retrieval and natural language processing, including cross-lingual in- formation retrieval (Lavrenko et al., 2002; Levow et al., 2005) and statistical machine translation (Och and Ney, 2003). Although parallel corpora have been used successfully for inducing bilin- gual lexicons for some languages (Och and Ney, 2003), these corpora are either too small or un- available for many language pairs. Consequently, mono-lingual approaches that rely on compara- ble instead of parallel corpora have been devel- oped (Fung and Yee, 1998; Koehn and Knight, 2002). These approaches work by mapping lan- guage pairs to a shared bilingual space and ex- tracting lexical items from that space. Bergsma and Van Durme (2011) showed that this bilingual space need not be linguistic in nature: they used labeled images from the Web to obtain bilingual lexical translation pairs based on the visual fea- tures of corresponding images. Local features are computed using SIFT (Lowe, 2004) and color his- tograms (Deselaers et al., 2008) and aggregated as bags of visual words (BOVW) (Sivic and Zisser- man, 2003) to get bilingual representations in a shared visual space. Their highest performance is obtained by combining these visual features with normalized edit distance, an orthographic similar- ity metric (Navarro, 2001). There are several advantages to having a vi- sual rather than a linguistic intermediate bilin- gual space: First, while labeled images are readily available for many languages through resources such as Google Images, language pairs that have sizeable comparable, let alone parallel, corpora are relatively scarce. Second, it has been found that meaning is often grounded in the perceptual system, and that the quality of semantic repre- sentations improves significantly when they are grounded in the visual modality (Silberer and La- pata, 2012; Bruni et al., 2014). Having an inter- …

Content truncated to 500 words