Models of Semantic Representation with Visual Attributes

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 572–582, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Models of Semantic Representation with Visual Attributes Carina Silberer, Vittorio Ferrari, Mirella Lapata School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB c.silberer@ed.ac.uk, vferrari@inf.ed.ac.uk, mlap@inf.ed.ac.uk Abstract We consider the problem of grounding the meaning of words in the physical world and focus on the visual modality which we represent by visual attributes. We create a new large-scale taxonomy of visual at- tributes covering more than 500 concepts and their corresponding 688K images. We use this dataset to train attribute classi- fiers and integrate their predictions with text-based distributional models of word meaning. We show that these bimodal models give a better fit to human word as- sociation data compared to amodal models and word representations based on hand- crafted norming data. 1 Introduction Recent years have seen increased interest in grounded language acquisition, where the goal is to extract representations of the meaning of nat- ural language tied to the physical world. The language grounding problem has assumed sev- eral guises in the literature such as semantic pars- ing (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Kate and Mooney, 2007; Lu et al., 2008; B¨orschinger et al., 2011), mapping nat- ural language instructions to executable actions (Branavan et al., 2009; Tellex et al., 2011), associ- ating simplified language to perceptual data such as images or video (Siskind, 2001; Roy and Pent- land, 2002; Gorniak and Roy, 2004; Yu and Bal- lard, 2007), and learning the meaning of words based on linguistic and perceptual input (Bruni et al., 2012b; Feng and Lapata, 2010; Johns and Jones, 2012; Andrews et al., 2009; Silberer and Lapata, 2012). In this paper we are concerned with the latter task, namely constructing perceptually grounded distributional models. The motivation for models that do not learn exclusively from text is twofold. From a cognitive perspective, there is mounting experimental evidence suggesting that our inter- action with the physical world plays an impor- tant role in the way we process language (Barsa- lou, 2008; Bornstein et al., 2004; Landau et al., 1998). From an engineering perspective, the abil- ity to learn representations for multimodal data has many practical applications including image re- trieval (Datta et al., 2008) and annotation (Chai and Hung, 2008), text illustration (Joshi et al., 2006), object and scene recognition (Lowe, 1999; Oliva and Torralba, 2007; Fei-Fei and Perona, 2005), and robot navigation (Tellex et al., 2011). One strand of research uses feature norms as a stand-in for sensorimotor experience (Johns and Jones, 2012; Andrews et al., 2009; Steyvers, 2010; Silberer and Lapata, 2012). Feature norms are ob- tained by asking native speakers to write down at- tributes they consider important in describing the meaning of a word. The attributes represent per- ceived physical and functional properties associ- ated with the referents of words. For example, apples are typically green or red, round, shiny, smooth, crunchy, tasty, and …

Content truncated to 500 words