Towards Robust Semantic Role Labeling

Proceedings of NAACL HLT 2007, pages 556–563, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics Towards Robust Semantic Role Labeling Sameer Pradhan BBN Technologies Cambridge, MA 02138 pradhan@bbn.com Wayne Ward, James H. Martin University of Colorado Boulder, CO 80303 {whw,martin}@colorado.edu Abstract Most research on semantic role labeling (SRL) has been focused on training and evaluating on the same corpus in order to develop the technology. This strategy, while appropriate for initiating research, can lead to over-training to the particular corpus. The work presented in this pa- per focuses on analyzing the robustness of an SRL system when trained on one genre of data and used to label a different genre. Our state-of-the-art semantic role labeling system, while performing well on WSJ test data, shows significant perfor- mance degradation when applied to data from the Brown corpus. We present a se- ries of experiments designed to investigate the source of this lack of portability. These experiments are based on comparisons of performance using PropBanked WSJ data and PropBanked Brown corpus data. Our results indicate that while syntactic parses and argument identification port relatively well to a new genre, argument classifica- tion does not. Our analysis of the reasons for this is presented and generally point to the nature of the more lexical/semantic features dominating the classification task and general structural features dominating the argument identification task. 1 Introduction Automatic, accurate and wide-coverage techniques that can annotate naturally occurring text with se- mantic argument structure play a key role in NLP applications such as Information Extraction (Sur- deanu et al., 2003; Harabagiu et al., 2005), Question Answering (Narayanan and Harabagiu, 2004) and Machine Translation (Boas, 2002; Chen and Fung, 2004). Semantic Role Labeling (SRL) is the pro- cess of producing such a markup. When presented with a sentence, a parser should, for each predicate in the sentence, identify and label the predicate’s se- mantic arguments. In recent work, a number of re- searchers have cast this problem as a tagging prob- lem and have applied various supervised machine learning techniques to it. On the Wall Street Jour- nal (WSJ) data, using correct syntactic parses, it is possible to achieve accuracies rivaling human inter- annotator agreement. However, the performance gap widens when information derived from automatic syntactic parses is used. So far, most of the work on SRL systems has been focused on improving the labeling performance on a test set belonging to the same genre of text as the training set. Both the Treebank on which the syntac- tic parser is trained and the PropBank on which the SRL systems are trained represent articles from the year 1989 of the WSJ. While all these systems per- form quite well on the WSJ test data, they show sig- nificant performance degradation (approximately 10 point drop in F-score) when applied to label test data that is different than the genre that WSJ represents (Pradhan et al., 2004; Carreras and M`arquez, 2005). 556 Surprisingly, it does not matter much whether the data is from …

Content truncated to 500 words