PropBank Comes of Age—Larger, Smarter, and more Diverse

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 278 - 288 July 14-15, 2022 ©2022 Association for Computational Linguistics PropBank Comes of Age—Larger, Smarter, and more Diverse Sameer Pradhan1,2, Julia Bonn3, Skatje Myers3, Kathryn Conger3. Tim O’Gorman4, James Gung5, Martha Palmer3 1University of Pennsylvania, Philadelphia, PA, 19104, 2cemantix.org 3University of Colorado, Boulder, CO 80303, 4Thorn, MI (thorn.org), 5Amazon AI, New York City, NY pradhan@cemantix.org, martha.palmer@colorado.edu Abstract This paper describes the evolution of the Prop- Bank approach to semantic role labeling over the last two decades. During this time the Prop- Bank frame files have been expanded to include non-verbal predicates such as adjectives, prepo- sitions and multi-word expressions. The num- ber of domains, genres and languages that have been PropBanked has also expanded greatly, creating an opportunity for much more chal- lenging and robust testing of the generalization capabilities of PropBank semantic role labeling systems. We also describe the substantial ef- fort that has gone into ensuring the consistency and reliability of the various annotated datasets and resources, to better support the training and evaluation of such systems. 1 Introduction Twenty years ago traditional statistical machine learning techniques were holding sway and suc- cessful stochastic syntactic parsing was on the rise. The availability of accurate syntactic parses opened the door to richer, deeper representations. The second Human Language Technology conference included a presentation on Adding Predicate Ar- gument Structure to the Penn Treebank and the Proposition Bank (PropBank) was born (Kingsbury and Palmer, 2002). Over the next few years, with the able guidance of a steering committee consist- ing of Ralph Weischedel, Mitch Marcus, Doug Appelt, Mark Villain and Ralph Grishman, the an- notation guidelines and the annotation continued to grow, with the end result of over 110,000 predicate argument structures pointing directly to syntactic nodes in the phrase structure syntax trees of the roughly 50,000 sentences of the Penn Treebank. The annotation of these structures was guided by a set of approximately 3300 Frame Files that pro- vided a verb specific set of semantic roles as the arguments for each verb. The substantial size of the data set and the consistency of the annotation gave rise to a flurry of popular semantic role la- beling systems and semantic role labeling shared tasks (Carreras and Màrquez, 2005; Surdeanu et al., 2008) that continue to this day. The Penn Treebank is entirely composed of Wall Street Journal articles, and annotation of additional data taken from the more diverse English genres of the Brown corpus allowed for out of domain testing, with predictable dismal results. Since that time, DARPA and NSF have funded substantial additional PropBank anno- tation, focusing on additional domains and genres for English, as well as additional languages such as Chinese, Arabic, Hindi and Urdu. The deep learn- ing revolution has not abated the interest in seman- tic role labeling performance, and the incorporation of PropBank Frame files into the Abstract Mean- ing Representation (AMR) Editor (Banarescu et al., 2013), to guide the labeling of …

Content truncated to 500 words