Analyzing Models for Semantic Role Assignment using Confusability

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 668–675, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Analyzing models for semantic role assignment using confusability Katrin Erk and Sebastian Padó Computational Linguistics Saarland University Saarbrücken, Germany {erk,pado}@coli.uni-sb.de Abstract We analyze models for semantic role assignment by defining a meta-model that abstracts over features and learning paradigms. This meta-model is based on the concept of role confusability, is de- fined in information-theoretic terms, and predicts that roles realized by less specific grammatical functions are more difficult to assign. We find that confusability is strongly correlated with the performance of classifiers based on syntactic features, but not for classifiers including semantic features. This indicates that syntactic fea- tures approximate a description of gram- matical functions, and that semantic fea- tures provide an independent second view on the data. 1 Introduction Semantic roles have become a focus of research in computational linguistics during the recent years. The driving force behind this interest is the prospect that semantic roles, as a shallow meaning represen- tation, can improve many NLP applications, while still being amenable to automatic analysis. The benefit of semantic roles has already been demon- strated for a number of tasks, among others for ma- chine translation (Boas, 2002), information extrac- tion (Surdeanu et al., 2003), and question answer- ing (Narayanan and Harabagiu, 2004). Robust and accurate automatic semantic role as- signment, a prerequisite for the wide-range use of semantic roles in NLP, has been investigated in a number of studies and shared tasks. Typically, role assignment has been modeled as a classification task, with models being estimated from large cor- pora (Gildea and Jurafsky, 2002; Moschitti, 2004; Xue and Palmer, 2004; Surdeanu et al., 2003; Prad- han et al., 2004; Litkowski, 2004; Carreras and Màrquez, 2005). Within this framework, there is a number of archi- tectural parameters which lend themselves to opti- mization: the machine learning framework, the fea- ture set, pre- and postprocessing, each of which has been investigated in the context of semantic role as- signment. The current paper concentrates on feature engineering, since the feature set is a pivotal com- ponent of any kind of machine learning system, and allows us to incorporate and test linguistic intuitions on the role assignment task. We approach feature engineering not by directly optimizing system performance. Instead, we pro- ceed by error analysis, like Pado and Boleda (2004). Our aim is to form a global hypothesis that explains the distribution of errors across classes. Insofar as the model does not contain model-specific infor- mation, following this methodology can provide a meta-model of a model family which abstracts over concrete features and over the learning paradigm. The concrete global hypothesis we test is: (1) All features of current models approximate a descrip- tion of grammatical functions, and the complete sys- tems approximate an assignment based on grammat- ical functions. (2) System performance for a given role depends on how easily it is confused with other …

Content truncated to 500 words