A Minimum Error Weighting Combination Strategy for Chinese Semantic Role Labeling

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1362–1370, Beijing, August 2010 A Minimum Error Weighting Combination Strategy for Chinese Semantic Role Labeling Tao Zhuang and Chengqing Zong National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences {tzhuang, cqzong}@nlpr.ia.ac.cn Abstract Many Semantic Role Labeling (SRL) combination strategies have been pro- posed and tested on English SRL task. But little is known about how much Chi- nese SRL can benefit from system combi- nation. And existing combination strate- gies trust each individual system’s output with the same confidence when merging them into a pool of candidates. In our ap- proach, we assign different weights to dif- ferent system outputs, and add a weighted merging stage to the conventional SRL combination architecture. We also pro- pose a method to obtain an appropriate weight for each system’s output by min- imizing some error function on the devel- opment set. We have evaluated our strat- egy on Chinese Proposition Bank data set. With our minimum error weighting strat- egy, the F1 score of the combined result achieves 80.45%, which is 1.12% higher than baseline combination method’s re- sult, and 4.90% higher than the best in- dividual system’s result. 1 Introduction In recent years, Chinese Semantic Role Labeling has received much research effort (Sun and Juraf- sky, 2004; Xue, 2008; Che et al., 2008; Ding and Chang, 2008; Sun et al., 2009; Li et al., 2009). And Chinese SRL is also included in CoNLL- 2009 shared task (Hajiˇc et al., 2009). On the data set used in (Xue, 2008), the F1 score of the SRL results using automatic syntactic analysis is still in low 70s (Xue, 2008; Che et al., 2008; Sun et al., 2009). As pointed out by Xue (Xue, 2008), the SRL errors are mainly caused by the errors in automatic syntactic analysis. In fact, Chinese SRL suffers from parsing errors even more than English SRL, because the state-of-the-art parser for Chinese is still not as good as that for En- glish. And previous research on English SRL shows that combination is a robust and effective method to alleviate SRL’s dependency on pars- ing results (M`arquez et al., 2005; Koomen et al., 2005; Pradhan et al., 2005; Surdeanu et al., 2007; Toutanova et al., 2008). However, the ef- fect of combination for Chinese SRL task is still unknown. This raises two questions at least: (1) How much can Chinese SRL benefit from combi- nation? (2) Can existing combination strategies be improved? All existing combination strate- gies trust each individual system’s output with the same confidence when putting them into a pool of candidates. But according to our intuition, dif- ferent systems have different performance. And the system that have better performance should be trusted with more confidence. We can use our prior knowledge about the combined systems to do a better combination. The observations above motivated the work in this paper. Instead of directly merging outputs with equal weights, different outputs are assigned different weights in …

Content truncated to 500 words