Robustness and Generalization of Role Sets: PropBank vs. VerbNet

Proceedings of ACL-08: HLT, pages 550–558, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Robustness and Generalization of Role Sets: PropBank vs. VerbNet Be˜nat Zapirain and Eneko Agirre IXA NLP Group University of the Basque Country {benat.zapirain,e.agirre}@ehu.es Llu´ıs M`arquez TALP Research Center Technical University of Catalonia lluism@lsi.upc.edu Abstract This paper presents an empirical study on the robustness and generalization of two alterna- tive role sets for semantic role labeling: Prop- Bank numbered roles and VerbNet thematic roles. By testing a state–of–the–art SRL sys- tem with the two alternative role annotations, we show that the PropBank role set is more robust to the lack of verb–specific semantic information and generalizes better to infre- quent and unseen predicates. Keeping in mind that thematic roles are better for application needs, we also tested the best way to generate VerbNet annotation. We conclude that tagging first PropBank roles and mapping into Verb- Net roles is as effective as training and tagging directly on VerbNet, and more robust for do- main shifts. 1 Introduction Semantic Role Labeling is the problem of analyzing clause predicates in open text by identifying argu- ments and tagging them with semantic labels indi- cating the role they play with respect to the verb. Such sentence–level semantic analysis allows to de- termine “who” did “what” to “whom”, “when” and “where”, and, thus, characterize the participants and properties of the events established by the predi- cates. This kind of semantic analysis is very inter- esting for a broad spectrum of NLP applications (in- formation extraction, summarization, question an- swering, machine translation, etc.), since it opens the door to exploit the semantic relations among lin- guistic constituents. The properties of the semantically annotated cor- pora available have conditioned the type of research and systems that have been developed so far. Prop- Bank (Palmer et al., 2005) is the most widely used corpus for training SRL systems, probably because it contains running text from the Penn Treebank cor- pus with annotations on all verbal predicates. Also, a few evaluation exercises on SRL have been con- ducted on this corpus in the CoNLL-2004 and 2005 conferences. However, a serious criticisms to the PropBank corpus refers to the role set it uses, which consists of a set of numbered core arguments, whose semantic translation is verb-dependent. While Arg0 and Arg1 are intended to indicate the general roles of Agent and Theme, other argument numbers do not generalize across verbs and do not correspond to general semantic roles. This fact might compro- mise generalization and portability of SRL systems, especially when the training corpus is small. More recently, a mapping from PropBank num- bered arguments into VerbNet thematic roles has been developed and a version of the PropBank cor- pus with thematic roles has been released (Loper et al., 2007). Thematic roles represent a compact set of verb-independent general roles widely used in lin- guistic theory (e.g., Agent, Theme, Patient, Recipi- ent, Cause, etc.). We foresee two advantages of us- ing such thematic roles. On …

Content truncated to 500 words