Word and Phrase Learning based on Prior Semantics

Proceedings of Recent Advances in Natural Language Processing, pages 616–621, Hissar, Bulgaria, 12-14 September 2011. Word and Phrase Learning based on Prior Semantics Amitabha Mukerjee and Nikhil Joshi Department of Computer Science and Engineering Indian Institute of Technology, Kanpur �UP, India) - 208016 �amit�joshins}@cse.iitk.ac.in Abstract Based on the evidences of prever- bal conceptual development in infants, we adopt semantics-first approach for word-learning. We first cluster several perceptual categories from a complex visual interaction. Using a surveillance traffic video, we a) identify the mov- ing objects by separating these from a static background, and b) group the similar appearances into clusters. The resulting models are found to be noisy approximations of traffic object cate- gories and motion actions. Next, we consider these models along with par- allel commentaries that describe the scene in free, unconstrained language. A bottom-up model of dynamic atten- tion is applied to identify objects in perceptual focus, which are mapped to words in co-temporaneous utterances. Using no language-specific knowledge such as syntax, we show the ability to learn words for the object classes and also for the motion actions. 1 Introduction The problem of word-learning primarily fo- cuses on mapping the linguistic representa- tion of a word to its semantics. In most of the attempts to learn the language to se- mantics mappings, semantic representations were often limited to the logical representa- tions (Zettlemoyer and Collins, 2005; H Al- shawi, 2011). However, the need of much richer semantic representations such as percp- tual schema (Barsalou, 1999), image schema (Mandler, 1992) is argued for grounding the meanings of words(Harnad, 1990). A num- ber of approaches have tried to construct such term-meaning associations from sensorimotor data (Steels and Kaplan, 2002; Gorniak and Roy, 2004; Roy and Pentland, 2002; Oates et al., 2000). However, these approaches used scenes with simple objects and constrained lin- guistic descriptions. Also, learning was guided by considerable feedback. In this work, we consider learning objects and interactions from a complex 3D-scene and mapping them to words and phrases from free, unconstrained language with full sentences de- scribing the scene. The key to handle referen- tial uncertainty (Siskind, 1996) is the visual saliency predicted using a bottom-up atten- tion model. The salient objects are then as- sociated with the co-occurring utterances in the narratives to learn the labels for the vi- sual concepts. Owing to evidences of preverbal conceptual development (Mandler, 1992), we adopt semantics-first approach (Yu and Bal- lard, 2004) where we learn visual semantics first and then discover appropriate word asso- ciated with it For learning objects and interactions, im- age sequences from a fixed camera, as typi- cally used in surveillance scenarios, are consid- ered. The stable patterns of background are first learned, and used to extract foreground blobs corresponding to the objects of inter- est. The object blobs are tracked across the frames and regions of occlusion are identified. Unoccluded object appearances are then pro- jected to a feature space based on the “Pyra- midal Histogram of visual Words” (PHOW) approach (Bosch …

Content truncated to 500 words