Jointly Learning Grounded Task Structures from Language Instruction and Visual Demonstration

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1482–1492, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Jointly Learning Grounded Task Structures from Language Instruction and Visual Demonstration Changsong Liu1*, Shaohua Yang1*, Sari Saba-Sadiya1, Nishant Shukla2, Yunzhong He2, Song-Chun Zhu2, and Joyce Y. Chai1 1Department of Computer Science and Engineering Michigan State University, East Lansing, MI 48824 2Center for Vision, Cognition, Learning, and Autonomy University of California, Los Angeles, CA 90095 {cliu,yangshao,sadiyasa,jchai}@cse.msu.edu {shukla,yunzhong}@cs.ucla.edu,sczhu@stat.ucla.edu Abstract To enable language-based communication and collaboration with cognitive robots, this paper presents an approach where an agent can learn task models jointly from language instruction and visual demonstration using an And-Or Graph (AoG) representation. The learned AoG captures a hierarchical task structure where linguistic labels (for language communication) are grounded to correspond- ing state changes from the physical environ- ment (for perception and action). Our em- pirical results on a cloth-folding domain have shown that, although state detection through visual processing is full of uncertainties and error prone, by a tight integration with lan- guage the agent is able to learn an effective AoG for task representation. The learned AoG can be further applied to infer and interpret on-going actions from new visual demonstra- tion using linguistic labels at different levels of granularity. 1 Introduction Given tremendous advances in robotics, computer vision, and natural language processing, a new gen- eration of cognitive robots have emerged that aim to collaborate with humans in joint tasks. To facili- tate natural and efficient communication with these physical agents, natural language processing will need to go beyond traditional symbolic representa- tions, but rather ground language to sensors (e.g., vi- sual perception) and actuators (e.g., lower-level con- trol systems) of physical agents. The internal task * The first two authors contributed equally to this paper. representation will need to capture both higher-level concepts (for language communication) and lower- level visual features (for perception and action). To address this need, we have developed an ap- proach on learning procedural tasks jointly from language instruction and visual demonstration. In particular, we use And-Or Graph (AoG), which has been used in many computer vision tasks and robotic applications (Zhao and Zhu, 2013; Li et al., 2016; Xiong et al., 2016), to represent a hier- archical task model that not only captures symbolic concepts (extracted from language instructions) but also the corresponding visual state changes from the physical environment (detected by computer vision algorithms). Different from previous works that ground lan- guage to perception (Liu et al., 2012; Matuszek et al., 2012; Kollar et al., 2013; Yu and Siskind, 2013; Yang et al., 2016), a key innovation in our frame- work is that language is no longer grounded just to perceived objects in the environment, but is further grounded to a hierarchical structure of state changes where the states are perceived from the environment during visual demonstration. The state of environ- ment is an important notion in robotic systems as the change of states drives planning for lower-level …

Content truncated to 500 words