Visual Information in Semantic Representation

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 91–99, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Visual Information in Semantic Representation Yansong Feng and Mirella Lapata School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB, UK Y.Feng-4@sms.ed.ac.uk, mlap@inf.ed.ac.uk Abstract The question of how meaning might be ac- quired by young children and represented by adult speakers of a language is one of the most debated topics in cognitive science. Existing semantic representation models are primarily amodal based on information provided by the linguistic input despite ample evidence indi- cating that the cognitive system is also sensi- tive to perceptual information. In this work we exploit the vast resource of images and associ- ated documents available on the web and de- velop a model of multimodal meaning repre- sentation which is based on the linguistic and visual context. Experimental results show that a closer correspondence to human data can be obtained by taking the visual modality into ac- count. 1 Introduction The representation and modeling of word mean- ing has been a central problem in cognitive science and natural language processing. Both disciplines are concerned with how semantic knowledge is ac- quired, organized, and ultimately used in language processing and understanding. A popular tradition of studying semantic representation has been driven by the assumption that word meaning can be learned from the linguistic environment. Words that are sim- ilar in meaning tend to behave similarly in terms of their distributions across different contexts. Se- mantic space models, among which Latent Semantic Analysis (LSA, Landauer and Dumais 1997) is per- haps known best, operationalize this idea by captur- ing word meaning quantitatively in terms of simple co-occurrence statistics. Each word w is represented by a k element vector reflecting the local distribu- tional context of w relative to k context words. More recently, topic models have been gaining ground as a more structured representation of word meaning. In contrast to more standard semantic space mod- els where word senses are conflated into a single representation, topic models assume that words ob- served in a corpus manifest some latent structure — word meaning is a probability distribution over a set of topics (corresponding to coarse-grained senses). Each topic is a probability distribution over words, and the content of the topic is reflected in the words to which it assigns high probability. Semantic space (and topic) models are extracted from real language corpora, and thus provide a direct means of investigating the influence of the statistics of language on semantic representation. They have been successful in explaining a wide range of be- havioral data — examples include lexical priming, deep dyslexia, text comprehension, synonym selec- tion, and human similarity judgments (see Landauer and Dumais 1997 and the references therein). They also underlie a large number of natural language processing (NLP) tasks including lexicon acquisi- tion, word sense discrimination, text segmentation and notably information retrieval. Despite their pop- ularity, these models …

Content truncated to 500 words