Is this a wampimuk? Cross-modal mapping between distributional semantics and the visual world

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1403–1414, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Is this a wampimuk? Cross-modal mapping between distributional semantics and the visual world Angeliki Lazaridou and Elia Bruni and Marco Baroni Center for Mind/Brain Sciences University of Trento {angeliki.lazaridou|elia.bruni|marco.baroni}@unitn.it Abstract Following up on recent work on estab- lishing a mapping between vector-based semantic embeddings of words and the visual representations of the correspond- ing objects from natural images, we first present a simple approach to cross-modal vector-based semantics for the task of zero-shot learning, in which an image of a previously unseen object is mapped to a linguistic representation denoting its word. We then introduce fast mapping, a challenging and more cognitively plausi- ble variant of the zero-shot task, in which the learner is exposed to new objects and the corresponding words in very limited linguistic contexts. By combining prior linguistic and visual knowledge acquired about words and their objects, as well as exploiting the limited new evidence avail- able, the learner must learn to associate new objects with words. Our results on this task pave the way to realistic simula- tions of how children or robots could use existing knowledge to bootstrap grounded semantic knowledge about new concepts. 1 Introduction Computational models of meaning that rely on corpus-extracted context vectors, such as LSA (Landauer and Dumais, 1997), HAL (Lund and Burgess, 1996), Topic Models (Griffiths et al., 2007) and more recent neural-network approaches (Collobert and Weston, 2008; Mikolov et al., 2013b) have successfully tackled a number of lex- ical semantics tasks, where context vector sim- ilarity highly correlates with various indices of semantic relatedness (Turney and Pantel, 2010). Given that these models are learned from natu- rally occurring data using simple associative tech- niques, various authors have advanced the claim that they might be also capturing some crucial as- pects of how humans acquire and use language (Landauer and Dumais, 1997; Lenci, 2008). However, the models induce the meaning of words entirely from their co-occurrence with other words, without links to the external world. This constitutes a serious blow to claims of cogni- tive plausibility in at least two respects. One is the grounding problem (Harnad, 1990; Searle, 1984). Irrespective of their relatively high per- formance on various semantic tasks, it is debat- able whether models that have no access to visual and perceptual information can capture the holis- tic, grounded knowledge that humans have about concepts. However, a possibly even more serious pitfall of vector models is lack of reference: natu- ral language is, fundamentally, a means to commu- nicate, and thus our words must be able to refer to objects, properties and events in the outside world (Abbott, 2010). Current vector models are purely language-internal, solipsistic models of meaning. Consider the very simple scenario in which visual information is being provided to an agent about the current state of the world, and the agent’s task is to determine the truth of a …

Content truncated to 500 words