Grounded Models of Semantic Representation

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1423–1433, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Grounded Models of Semantic Representation Carina Silberer and Mirella Lapata Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB c.silberer@ed.ac.uk, mlap@inf.ed.ac.uk Abstract A popular tradition of studying semantic rep- resentation has been driven by the assump- tion that word meaning can be learned from the linguistic environment, despite ample ev- idence suggesting that language is grounded in perception and action. In this paper we present a comparative study of models that represent word meaning based on linguistic and perceptual data. Linguistic information is approximated by naturally occurring corpora and sensorimotor experience by feature norms (i.e., attributes native speakers consider impor- tant in describing the meaning of a word). The models differ in terms of the mechanisms by which they integrate the two modalities. Ex- perimental results show that a closer corre- spondence to human data can be obtained by uncovering latent information shared among the textual and perceptual modalities rather than arriving at semantic knowledge by con- catenating the two. 1 Introduction Distributional models of lexical semantics have seen considerable success at accounting for a wide range of behavioral data in tasks involving semantic cog- nition (Landauer and Dumais, 1997; Griffiths et al., 2007). These models have also enjoyed last- ing popularity in natural language processing. Ex- amples involve information retrieval (Salton et al., 1975), word sense discrimination (Sch¨utze, 1998), text segmentation (Choi et al., 2001), and numerous studies of lexicon acquisition (Grefenstette, 1994; Lin, 1998). Despite their widespread use, distribu- tional models have been criticized as “disembodied” in that they learn exclusively from linguistic infor- mation but are not grounded in perception and ac- tion (Perfetti, 1998; Barsalou, 1999; Glenberg and Kaschak, 2002). This lack of grounding contrasts with many ex- perimental studies suggesting that word meaning is acquired not only from exposure to the linguistic environment but also from our interaction with the physical world (Landau et al., 1998; Bornstein et al., 2004). Beyond language acquisition, there is consid- erable evidence across both behavioral experiments and neuroimaging studies that the perceptual asso- ciates of words play an important role in language processing (for a review see Barsalou (2008)). It is thus no surprise that recent years have wit- nessed the emergence of perceptually grounded dis- tributional models. An important question in the for- mulation of such models concerns the provenance of perceptual information. A few models use fea- ture norms as a proxy for sensorimotor experience (Howell et al., 2005; Andrews et al., 2009; Steyvers, 2010; Johns and Jones, 2012). These are obtained by asking native speakers to write down attributes they consider important in describing the meaning of a word. The attributes represent perceived phys- ical and functional properties associated with the referents of words. For example, apples are typi- cally green or red, round, shiny, smooth, …

Content truncated to 500 words