Physical Causality of Action Verbs in Grounded Language Understanding

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1814–1824, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Physical Causality of Action Verbs in Grounded Language Understanding Qiaozi Gao† Malcolm Doering‡∗ Shaohua Yang† Joyce Y. Chai† †Computer Science and Engineering, Michigan State University, East Lansing, MI, USA ‡Department of Systems Innovation, Osaka University, Toyonaka, Osaka, Japan {gaoqiaoz, yangshao, jchai}@msu.edu doering.malcolm.robert@irl.sys.es.osaka-u.ac.jp Abstract Linguistics studies have shown that action verbs often denote some Change of State (CoS) as the result of an action. However, the causality of action verbs and its poten- tial connection with the physical world has not been systematically explored. To ad- dress this limitation, this paper presents a study on physical causality of action verbs and their implied changes in the physi- cal world. We first conducted a crowd- sourcing experiment and identified eigh- teen categories of physical causality for action verbs. For a subset of these cat- egories, we then defined a set of detec- tors that detect the corresponding change from visual perception of the physical en- vironment. We further incorporated phys- ical causality modeling and state detec- tion in grounded language understanding. Our empirical studies have demonstrated the effectiveness of causality modeling in grounding language to perception. 1 Introduction Linguistics studies have shown that action verbs often denote some change of state (CoS) as the result of an action, where the change of state of- ten involves an attribute of the direct object of the verb (Hovav and Levin, 2010). For example, the result of “slice a pizza” is that the state of the ob- ject (pizza) changes from one big piece to several smaller pieces. This change of state can be per- ceived from the physical world. In Artificial Intel- ligence (Russell and Norvig, 2010), decades of re- search on planning, for example, back to the early days of the STRIPS planner (Fikes and Nilsson, ∗This work was conducted at Michigan State University where the author received his MS degree. 1971), have defined action schemas to capture the change of state caused by a given action. Based on action schemas, planning algorithms can be ap- plied to find a sequence of actions to achieve a goal state (Ghallab et al., 2004). The state of the phys- ical world is a very important notion and chang- ing the state becomes a driving force for agents’ actions. Thus, motivated by linguistic literature on action verbs and AI literature on action repre- sentations, in our view, modeling change of phys- ical state for action verbs, in other words, physical causality, can better connect language to the phys- ical world. Although this kind of physical causality has been described in linguistic studies (Hovav and Levin, 2010), a detailed account of potential causality that could be denoted by an action verb is lacking. For example, in VerbNet (Schuler, 2005) the semantic representation for various verbs may indicate that a change of state is involved, but it does not provide the specifics associated with …

Content truncated to 500 words