The Influence of Argument Structure on Semantic Role Assignment

The Influence of Argument Structure on Semantic Role Assignment Sebastian Padó SALSA Dept. of Computational Linguistics Saarland University Saarbrücken pado@coli.uni-sb.de Gemma Boleda GLiCom Dept. of Translation and Interpreting Pompeu Fabra University Barcelona gemma.boleda@upf.edu Abstract We present a data and error analysis for semantic role labelling. In a first experiment, we build a generic statis- tical model for semantic role assignment in the FrameNet paradigm and show that there is a high variance in perfor- mance across frames. The main hypothesis of our paper is that this variance is to a large extent a result of differ- ences in the underlying argument structure of the pred- icates in different frames. In a second experiment, we show that frame uniformity, which measures argument structure variation, correlates well with the performance figures, effectively explaining the variance. 1 Introduction Recent years have witnessed growing interest in corpora with semantic annotation, especially on the semantic role (or argument structure) level. A number of projects are working on producing such corpora through manual annotation, among which are FrameNet (Baker et al., 1998), the Prague Dependency Treebank (Hajiˇcová, 1998), Prop- Bank (Kingsbury et al., 2002), and SALSA (Erk et al., 2003). For semantic role annotation to be widely useful for NLP, however, robust and accurate methods for automatic semantic role assignment are necessary. Starting with Gildea and Jurafsky (2000), a num- ber of studies have developed (almost exclusively statistical) models of this task, e.g. Thompson et al. (2003) and Fleischman et al. (2003). This year (2004), semantic role labelling served as the shared task at two conferences, CoNLL1 and SENSEVAL2. However, almost all studies have concentrated on the technical aspects of the models – identifying in- formative feature sets and suitable statistical frame- works – with the goal of optimising the performance of the models on the complete dataset. The only study we are aware of with a more detailed evalu- ation is Fleischman et al. (2003), who nevertheless come to the conclusion that either “new features”, 1http://www.lsi.upc.es/~conll04st/ 2http://www.clres.com/SensSemRoles.html “more data”, or “more sophisticated models” are needed. The present study is a fi rst step in pursuing the third alternative, presenting a data and error anal- ysis for semantic role assignment in the FrameNet paradigm. We fi rst build two different, generic sta- tistical models for semantic role assignment, which are fairly representative for the span of models in- vestigated in the literature. A frame-wise evalua- tion shows that the models exhibit a large variance in performance across frames. Our hypothesis is that this variance is to a large extent caused by differences in the underlying ar- gument structure of the predicates: Frames which are less uniform, i.e. whose predicates have a more heterogeneous mapping between semantic roles and syntactic functions, are more diffi cult to label auto- matically. In order to put this hypothesis, which is intuitively very plausible, on a a fi rm empirical foot- ing, we investigate the relationship between frame uniformity and the variance in the data and show that the two variables …

Content truncated to 500 words