Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 36–45, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics Douwe Kiela∗ University of Cambridge Computer Laboratory douwe.kiela@cl.cam.ac.uk L´eon Bottou Microsoft Research New York leon@bottou.org Abstract We construct multi-modal concept repre- sentations by concatenating a skip-gram linguistic representation vector with a vi- sual concept representation vector com- puted using the feature extraction layers of a deep convolutional neural network (CNN) trained on a large labeled object recognition dataset. This transfer learn- ing approach brings a clear performance gain over features based on the traditional bag-of-visual-word approach. Experimen- tal results are reported on the WordSim353 and MEN semantic relatedness evaluation tasks. We use visual features computed us- ing either ImageNet or ESP Game images. 1 Introduction Recent works have shown that multi-modal se- mantic representation models outperform uni- modal linguistic models on a variety of tasks, in- cluding modeling semantic relatedness and pre- dicting compositionality (Feng and Lapata, 2010; Leong and Mihalcea, 2011; Bruni et al., 2012; Roller and Schulte im Walde, 2013; Kiela et al., 2014). These results were obtained by combin- ing linguistic feature representations with robust visual features extracted from a set of images as- sociated with the concept in question. This extrac- tion of visual features usually follows the popular computer vision approach consisting of comput- ing local features, such as SIFT features (Lowe, 1999), and aggregating them as bags of visual words (Sivic and Zisserman, 2003). Meanwhile, deep transfer learning techniques have gained considerable attention in the com- puter vision community. First, a deep convolu- tional neural network (CNN) is trained on a large ∗ This work was carried out while Douwe Kiela was an intern at Microsoft Research, New York. labeled dataset (Krizhevsky et al., 2012). The convolutional layers are then used as mid-level feature extractors on a variety of computer vi- sion tasks (Oquab et al., 2014; Girshick et al., 2013; Zeiler and Fergus, 2013; Donahue et al., 2014). Although transferring convolutional net- work features is not a new idea (Driancourt and Bottou, 1990), the simultaneous availability of large datasets and cheap GPU co-processors has contributed to the achievement of considerable performance gains on a variety computer vision benchmarks: “SIFT and HOG descriptors pro- duced big performance gains a decade ago, and now deep convolutional features are providing a similar breakthrough” (Razavian et al., 2014). This work reports on results obtained by using CNN-extracted features in multi-modal semantic representation models. These results are interest- ing in several respects. First, these superior fea- tures provide the opportunity to increase the per- formance gap achieved by augmenting linguistic features with multi-modal features. Second, this increased performance confirms that the multi- modal performance improvement results from the information contained in the images and not the information used to select which images to use to represent a concept. Third, our evaluation re- veals an intriguing property of the CNN-extracted features. Finally, since …

Content truncated to 500 words