Combining Language and Vision with a Multimodal Skip-gram Model

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 153–163, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Combining Language and Vision with a Multimodal Skip-gram Model Angeliki Lazaridou Nghia The Pham Marco Baroni Center for Mind/Brain Sciences University of Trento {angeliki.lazaridou|thenghia.pham|marco.baroni}@unitn.it Abstract We extend the SKIP-GRAM model of Mikolov et al. (2013a) by taking visual information into account. Like SKIP-GRAM, our multimodal models (MMSKIP-GRAM) build vector-based word representations by learning to predict linguistic contexts in text corpora. However, for a restricted set of words, the models are also exposed to visual representations of the objects they denote (extracted from natural images), and must predict linguistic and visual features jointly. The MMSKIP-GRAM mod- els achieve good performance on a variety of semantic benchmarks. Moreover, since they propagate visual information to all words, we use them to improve image labeling and re- trieval in the zero-shot setup, where the test concepts are never seen during model training. Finally, the MMSKIP-GRAM models discover intriguing visual properties of abstract words, paving the way to realistic implementations of embodied theories of meaning. 1 Introduction Distributional semantic models (DSMs) derive vector-based representations of meaning from pat- terns of word co-occurrence in corpora. DSMs have been very effectively applied to a variety of seman- tic tasks (Clark, 2015; Mikolov et al., 2013b; Turney and Pantel, 2010). However, compared to human semantic knowledge, these purely textual models, just like traditional symbolic AI systems (Harnad, 1990; Searle, 1984), are severely impoverished, suf- fering of lack of grounding in extra-linguistic modal- ities (Glenberg and Robertson, 2000). This observa- tion has led to the development of multimodal dis- tributional semantic models (MDSMs) (Bruni et al., 2014; Feng and Lapata, 2010; Silberer and Lapata, 2014), that enrich linguistic vectors with perceptual information, most often in the form of visual fea- tures automatically induced from image collections. MDSMs outperform state-of-the-art text-based approaches, not only in tasks that directly require access to visual knowledge (Bruni et al., 2012), but also on general semantic benchmarks (Bruni et al., 2014; Silberer and Lapata, 2014). However, current MDSMs still have a number of drawbacks. First, they are generally constructed by first separately building linguistic and visual representations of the same concepts, and then merging them. This is ob- viously very different from how humans learn about concepts, by hearing words in a situated perceptual context. Second, MDSMs assume that both linguis- tic and visual information is available for all words, with no generalization of knowledge across modal- ities. Third, because of this latter assumption of full linguistic and visual coverage, current MDSMs, paradoxically, cannot be applied to computer vision tasks such as image labeling or retrieval, since they do not generalize to images or words beyond their training set. We introduce the multimodal skip-gram models, two new MDSMs that address all the issues above. The models build upon the very effective skip-gram approach of Mikolov et al. (2013a), that constructs vector representations by learning, …

Content truncated to 500 words