Open-Domain Semantic Role Labeling by Modeling Word Spans

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 968–978, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Open-Domain Semantic Role Labeling by Modeling Word Spans Fei Huang Temple University 1805 N. Broad St. Wachman Hall 318 fei.huang@temple.edu Alexander Yates Temple University 1805 N. Broad St. Wachman Hall 303A yates@temple.edu Abstract Most supervised language processing sys- tems show a significant drop-off in per- formance when they are tested on text that comes from a domain significantly different from the domain of the training data. Semantic role labeling techniques are typically trained on newswire text, and in tests their performance on fiction is as much as 19% worse than their perfor- mance on newswire text. We investigate techniques for building open-domain se- mantic role labeling systems that approach the ideal of a train-once, use-anywhere system. We leverage recently-developed techniques for learning representations of text using latent-variable language mod- els, and extend these techniques to ones that provide the kinds of features that are useful for semantic role labeling. In exper- iments, our novel system reduces error by 16% relative to the previous state of the art on out-of-domain text. 1 Introduction In recent semantic role labeling (SRL) competi- tions such as the shared tasks of CoNLL 2005 and CoNLL 2008, supervised SRL systems have been trained on newswire text, and then tested on both an in-domain test set (Wall Street Journal text) and an out-of-domain test set (fiction). All sys- tems tested on these datasets to date have exhib- ited a significant drop-off in performance on the out-of-domain tests, often performing 15% worse or more on the fiction test sets. Yet the baseline from CoNLL 2005 suggests that the fiction texts are actually easier than the newswire texts. Such observations expose a weakness of current super- vised natural language processing (NLP) technol- ogy for SRL: systems learn to identify semantic roles for the subset of language contained in the training data, but are not yet good at generalizing to language that has not been seen before. We aim to build an open-domain supervised SRL system; that is, one whose performance on out-of-domain tests approaches the same level of performance as that of state-of-the-art systems on in-domain tests. Importantly, an open-domain sys- tem must not use any new labeled data beyond what is included in the original training text when running on a new domain. This allows the sys- tem to be ported to any new domain without any manual effort. In particular, it ought to apply to arbitrary Web documents, which are drawn from a huge variety of domains. Recent theoretical and empirical evidence sug- gests that the fault for poor performance on out-of- domain tests lies with the representations, or sets of features, traditionally used in supervised NLP. Building on recent efforts in domain adaptation, we develop unsupervised techniques for learning new representations of text. Using latent-variable language models, we learn representations of texts that provide novel kinds of features to our su- pervised …

Content truncated to 500 words