Comparing Data Sources and Architectures for Deep Visual Representation Learning in Semantics

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 447–456, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Comparing Data Sources and Architectures for Deep Visual Representation Learning in Semantics Douwe Kiela, Anita L. Ver˝o and Stephen Clark Computer Laboratory University of Cambridge douwe.kiela,alv34,stephen.clark@cl.cam.ac.uk Abstract Multi-modal distributional models learn grounded representations for improved performance in semantics. Deep visual representations, learned using convolutional neural networks, have been shown to achieve particularly high performance. In this study, we systematically compare deep visual representation learning techniques, exper- imenting with three well-known network architectures. In addition, we explore the various data sources that can be used for retrieving relevant images, showing that images from search engines perform as well as, or better than, those from manually crafted resources such as ImageNet. Furthermore, we explore the optimal number of images and the multi-lingual applicability of multi-modal semantics. We hope that these findings can serve as a guide for future research in the field. 1 Introduction Multi-modal distributional semantics addresses the fact that text-based semantic models, which rep- resent word meanings as a distribution over other words (Turney and Pantel, 2010; Clark, 2015), suf- fer from the grounding problem (Harnad, 1990). Re- cent work has shown that this theoretical motivation can be successfully exploited for practical gain. In- deed, multi-modal representation learning leads to improvements over language-only models in a range of tasks, including modelling semantic similarity and relatedness (Bruni et al., 2014; Silberer and La- pata, 2014; Kiela and Bottou, 2014; Lazaridou et al., 2015), improving lexical entailment (Kiela et al., 2015a), predicting compositionality (Roller and Schulte im Walde, 2013), bilingual lexicon induc- tion (Bergsma and Van Durme, 2011), selectional preference prediction (Bergsma and Goebel, 2011), linguistic ambiguity resolution (Berzak et al., 2015), visual information retrieval (Bulat et al., 2016) and metaphor identification (Shutova et al., 2016). Most multi-modal semantic models tend to rely on raw images as the source of perceptual input. Many data sources have been tried, ranging from image search engines to photo sharing websites to manually crafted resources. Images are retrieved for a given target word if they are ranked highly, have been tagged, or are otherwise associated with the tar- get word(s) in the data source. Traditionally, representations for images were learned through bag-of-visual words (Sivic and Zis- serman, 2003), using SIFT-based local feature de- scriptors (Lowe, 2004). Kiela and Bottou (2014) showed that transferring representations from deep convolutional neural networks (ConvNets) yield much better performance than bag-of-visual-words in multi-modal semantics. ConvNets (LeCun et al., 1998) have become very popular in recent years: they are now the dominant approach for almost all recognition and detection tasks in the com- puter vision community (LeCun et al., 2015), ap- proaching or even exceeding human performance in some cases (Weyand et al., 2016). The work by Alex Krizhevsky et al. (2012), which won the Im- ageNet Large Scale Visual Recognition Challenge (ILSVRC) (Russakovsky et al., 2015) in 2012, has played an important role in bringing convolutional 447 …

Content truncated to 500 words