Maximum Metric Score Training for Coreference Resolution

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1308–1316, Beijing, August 2010 Maximum Metric Score Training for Coreference Resolution Shanheng Zhao and Hwee Tou Ng Department of Computer Science National University of Singapore {zhaosh,nght}@comp.nus.edu.sg Abstract A large body of prior research on coref- erence resolution recasts the problem as a two-class classification problem. How- ever, standard supervised machine learn- ing algorithms that minimize classifica- tion errors on the training instances do not always lead to maximizing the F-measure of the chosen evaluation metric for coref- erence resolution. In this paper, we pro- pose a novel approach comprising the use of instance weighting and beam search to maximize the evaluation metric score on the training corpus during training. Ex- perimental results show that this approach achieves significant improvement over the state-of-the-art. We report results on stan- dard benchmark corpora (two MUC cor- pora and three ACE corpora), when evalu- ated using the link-based MUC metric and the mention-based B-CUBED metric. 1 Introduction Coreference resolution refers to the process of determining whether two or more noun phrases (NPs) in a text refer to the same entity. Suc- cessful coreference resolution benefits many nat- ural language processing tasks. In the literature, most prior work on coreference resolution recasts the problem as a two-class classification problem. Machine learning-based classifiers are applied to determine whether a candidate anaphor and a po- tential antecedent are coreferential (Soon et al., 2001; Ng and Cardie, 2002b). A large body of prior research on corefer- ence resolution follows the same process: dur- ing training, they apply standard supervised ma- chine learning algorithms to minimize the number of misclassified training instances; during testing, they maximize either the local or the global proba- bility of the coreferential relation assignments ac- cording to the specific chosen resolution method. However, minimizing the number of misclas- sified training instances during training does not guarantee maximizing the F-measure of the cho- sen evaluation metric for coreference resolution. First of all, coreference is a rare relation. There are far fewer positive training instances than neg- ative ones. Simply minimizing the number of mis- classified training instances is suboptimal and fa- vors negative training instances. Secondly, evalu- ation metrics for coreference resolution are based on global assignments. Not all errors have the same impact on the metric score. Furthermore, the extracted training instances are not equally easy to be classified. In this paper, we propose a novel approach comprising the use of instance weighting and beam search to address the above issues. Our pro- posed maximum metric score training (MMST) approach performs maximization of the chosen evaluation metric score on the training corpus dur- ing training. It iteratively assigns higher weights to the hard-to-classify training instances. The out- put of training is a standard classifier. Hence, during testing, MMST is faster than approaches which optimize the assignment of coreferential re- lations during testing. Experimental results show that MMST achieves significant improvements over the baselines. Unlike most of the previous work, we report improved …

Content truncated to 500 words