Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World

Transactions of the Association for Computational Linguistics, 1 (2013) 193–206. Action Editor: Jason Eisner. Submitted 10/2012; Revised 3/2013; Published 5/2013. c⃝2013 Association for Computational Linguistics. Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World Jayant Krishnamurthy Computer Science Department Carnegie Mellon University jayantk@cs.cmu.edu Thomas Kollar Computer Science Department Carnegie Mellon University tkollar@andrew.cmu.edu Abstract This paper introduces Logical Semantics with Perception (LSP), a model for grounded lan- guage acquisition that learns to map natu- ral language statements to their referents in a physical environment. For example, given an image, LSP can map the statement “blue mug on the table” to the set of image seg- ments showing blue mugs on tables. LSP learns physical representations for both cate- gorical (“blue,” “mug”) and relational (“on”) language, and also learns to compose these representations to produce the referents of en- tire statements. We further introduce a weakly supervised training procedure that estimates LSP’s parameters using annotated referents for entire statements, without annotated ref- erents for individual words or the parse struc- ture of the statement. We perform experiments on two applications: scene understanding and geographical question answering. We find that LSP outperforms existing, less expressive models that cannot represent relational lan- guage. We further find that weakly supervised training is competitive with fully supervised training while requiring significantly less an- notation effort. 1 Introduction Learning the mapping from natural language to physical environments is a central problem for natu- ral language semantics. Understanding this mapping is necessary to enable natural language interactions with robots and other embodied systems. For exam- ple, for an autonomous robot to understand the sen- tence “The blue mug is on the table,” it must be able to identify (1) the objects in its environment corre- sponding to “blue mug” and “table,” and (2) the ob- jects which participate in the spatial relation denoted by “on.” If the robot can successfully identify these objects, it understands the meaning of the sentence. The problem of learning to map from natural lan- guage expressions to their referents in an environ- ment is known as grounded language acquisition. In embodied settings, environments consist of raw sensor data – for example, an environment could be an image collected from a robot’s camera. In such applications, grounded language acquisition has two subproblems: parsing, learning the compositional structure of natural language; and perception, learn- ing the environmental referents of individual words. Acquiring both kinds of knowledge is necessary to understand novel language in novel environments. Unfortunately, perception is often ignored in work on language acquisition. Other variants of grounded language acquisition eliminate the need for percep- tion by assuming access to a logical representation of the environment (Zettlemoyer and Collins, 2005; Wong and Mooney, 2006; Matuszek et al., 2010; Chen and Mooney, 2011; Liang et al., 2011). The existing work which has jointly addressed both pars- ing and perception has significant drawbacks, in- cluding: (1) fully supervised models requiring large amounts of manual annotation and (2) limited se- mantic representations (Kollar …

Content truncated to 500 words