Learning the Semantics of Manipulation Action

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 676–686, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Learning the Semantics of Manipulation Action Yezhou Yang† and Yiannis Aloimonos† and Cornelia Ferm¨uller† and Eren Erdal Aksoy‡ † UMIACS, University of Maryland, College Park, MD, USA {yzyang, yiannis, fer}@umiacs.umd.edu ‡ Karlsruhe Institute of Technology, Karlsruhe, Germany eren.aksoy@kit.edu Abstract In this paper we present a formal compu- tational framework for modeling manip- ulation actions. The introduced formal- ism leads to semantics of manipulation ac- tion and has applications to both observ- ing and understanding human manipula- tion actions as well as executing them with a robotic mechanism (e.g. a humanoid robot). It is based on a Combinatory Cat- egorial Grammar. The goal of the intro- duced framework is to: (1) represent ma- nipulation actions with both syntax and se- mantic parts, where the semantic part em- ploys λ-calculus; (2) enable a probabilis- tic semantic parsing schema to learn the lambda-calculus representation of manip- ulation action from an annotated action corpus of videos; (3) use (1) and (2) to de- velop a system that visually observes ma- nipulation actions and understands their meaning while it can reason beyond ob- servations using propositional logic and axiom schemata. The experiments con- ducted on a public available large manip- ulation action dataset validate the theoret- ical framework and our implementation. 1 Introduction Autonomous robots will need to learn the actions that humans perform. They will need to recognize these actions when they see them and they will need to perform these actions themselves. This re- quires a formal system to represent the action se- mantics. This representation needs to store the se- mantic information about the actions, be encoded in a machine readable language, and inherently be in a programmable fashion in order to enable rea- soning beyond observation. A formal represen- tation of this kind has a variety of other appli- cations such as intelligent manufacturing, human robot collaboration, action planning and policy de- sign, etc. In this paper, we are concerned with manipula- tion actions, that is actions performed by agents (humans or robots) on objects, resulting in some physical change of the object. However most of the current AI systems require manually defined semantic rules. In this work, we propose a com- putational linguistics framework, which is based on probabilistic semantic parsing with Combina- tory Categorial Grammar (CCG), to learn manip- ulation action semantics (lexicon entries) from an- notations. We later show that this learned lexicon is able to make our system reason about manipu- lation action goals beyond just observation. Thus the intelligent system can not only imitate human movements, but also imitate action goals. Understanding actions by observation and exe- cuting them are generally considered as dual prob- lems for intelligent agents. The sensori-motor bridge connecting the two tasks is essential, and a great amount of attention in AI, Robotics as well as Neurophysiology has …

Content truncated to 500 words