Grounded Semantic Role Labeling

Proceedings of NAACL-HLT 2016, pages 149–159, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Grounded Semantic Role Labeling Shaohua Yang1, Qiaozi Gao1, Changsong Liu1, Caiming Xiong2, Song-Chun Zhu3, and Joyce Y. Chai1 1Department of Computer Science and Engineering, Michigan State University, East Lansing, MI 48824 2Metamind, Palo Alto, CA 94301 3Center for Vision, Cognition, Learning, and Autonomy, University of California, Los Angeles, CA 90095 {yangshao,gaoqiaoz,cliu,jchai}@cse.msu.edu cmxiong@metamind.io, sczhu@stat.ucla.edu Abstract Semantic Role Labeling (SRL) captures se- mantic roles (or participants) such as agent, patient, and theme associated with verbs from the text. While it provides important in- termediate semantic representations for many traditional NLP tasks (such as information ex- traction and question answering), it does not capture grounded semantics so that an arti- ficial agent can reason, learn, and perform the actions with respect to the physical envi- ronment. To address this problem, this pa- per extends traditional SRL to grounded SRL where arguments of verbs are grounded to participants of actions in the physical world. By integrating language and vision process- ing through joint inference, our approach not only grounds explicit roles, but also grounds implicit roles that are not explicitly mentioned in language descriptions. This paper describes our empirical results and discusses challenges and future directions. 1 Introduction Linguistic studies capture semantics of verbs by their frames of thematic roles (also referred to as semantic roles or verb arguments) (Levin, 1993). For example, a verb can be characterized by agent (i.e., the animator of the action) and patient (i.e., the object on which the action is acted upon), and other roles such as instrument, source, destination, etc. Given a verb frame, the goal of Semantic Role Labeling (SRL) is to identify lin- guistic entities from the text that serve different the- matic roles (Palmer et al., 2005; Gildea and Jurafsky, The$woman$takes$out$a$cucumber$from$the$ refrigerator.$ !Predicate:!“takes$out”:$track$1$ !Agent:$‘’The$woman’’$:$track$2$ !Pa.ent:$‘’a$cucumber’’$:$track$3$ !Source:$‘’from$the$refrigerator’’$:$ track$4$ !Des.na.on:$‘’$‘’$:$track$5$ $ Figure 1: An example of grounded semantic role labeling for the sentence the woman takes out a cucumber from the refrig- erator. The left hand side shows three frames of a video clip with the corresponding language description. The objects in the bounding boxes are tracked and each track has a unique identi- fier. The right hand side shows the grounding results where each role including the implicit role (destination) is grounded to a track id. 2002; Collobert et al., 2011; Zhou and Xu, 2015). For example, given the sentence the woman takes out a cucumber from the refrigerator., takes out is the main verb (also called predicate); the noun phrase the woman is the agent of this action; a cu- cumber is the patient; and the refrigerator is the source. SRL captures important semantic representations for actions associated with verbs, which have shown beneficial for a variety of applications such as infor- mation extraction (Emanuele et al., 2013) and ques- tion answering (Shen and Lapata, 2007). However, the traditional SRL is not targeted to represent verb semantics that are grounded to the physical world so that artificial …

Content truncated to 500 words