Grounded Language Learning from Video Described with Sentences

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 53–63, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Grounded Language Learning from Video Described with Sentences Haonan Yu and Jeffrey Mark Siskind Purdue University School of Electrical and Computer Engineering 465 Northwestern Ave. West Lafayette, IN 47907-2035 USA haonan@haonanyu.com, qobi@purdue.edu Abstract We present a method that learns repre- sentations for word meanings from short video clips paired with sentences. Un- like prior work on learning language from symbolic input, our input consists of video of people interacting with multiple com- plex objects in outdoor environments. Un- like prior computer-vision approaches that learn from videos with verb labels or im- ages with noun labels, our labels are sen- tences containing nouns, verbs, preposi- tions, adjectives, and adverbs. The cor- respondence between words and concepts in the video is learned in an unsupervised fashion, even when the video depicts si- multaneous events described by multiple sentences or when different aspects of a single event are described with multiple sentences. The learned word meanings can be subsequently used to automatically generate description of new video. 1 Introduction People learn language through exposure to a rich perceptual context. Language is grounded by mapping words, phrases, and sentences to mean- ing representations referring to the world. Siskind (1996) has shown that even with referential un- certainty and noise, a system based on cross- situational learning can robustly acquire a lexicon, mapping words to word-level meanings from sen- tences paired with sentence-level meanings. How- ever, it did so only for symbolic representations of word- and sentence-level meanings that were not perceptually grounded. An ideal system would not require detailed word-level labelings to acquire word meanings from video but rather could learn language in a largely unsupervised fashion, just as a child does, from video paired with sentences. There has been recent research on grounded lan- guage learning. Roy (2002) pairs training sen- tences with vectors of real-valued features ex- tracted from synthesized images which depict 2D blocks-world scenes, to learn a specific set of fea- tures for adjectives, nouns, and adjuncts. Yu and Ballard (2004) paired training images containing multiple objects with spoken name candidates for the objects to find the correspondence between lexical items and visual features. Dominey and Boucher (2005) paired narrated sentences with symbolic representations of their meanings, au- tomatically extracted from video, to learn object names, spatial-relation terms, and event names as a mapping from the grammatical structure of a sentence to the semantic structure of the associated meaning representation. Chen and Mooney (2008) learned the language of sportscasting by deter- mining the mapping between game commentaries and the meaning representations output by a rule- based simulation of the game. Kwiatkowski et al. (2012) present an approach that learns Montague- grammar representations of word meanings to- gether with a combinatory categorial grammar (CCG) from child-directed sentences paired with first-order formulas that represent their meaning. Although most of these methods succeed in learning word meanings from sentential …

Content truncated to 500 words