Augmenting ensemble classification for Word Sense Disambiguation with a kernel PCA model

Augmenting Ensemble Classification for Word Sense Disambiguation with a Kernel PCA Model Marine CARPUAT Weifeng SU Dekai WU1 marine@cs.ust.hk weifeng@cs.ust.hk dekai@cs.ust.hk Human Language Technology Center HKUST Department of Computer Science University of Science and Technology, Clear Water Bay, Hong Kong Abstract The HKUST word sense disambiguation systems benefit from a new nonlinear Kernel Principal Component Analysis (KPCA) based disambigua- tion technique. We discuss and analyze results from the Senseval-3 English, Chinese, and Multi- lingual Lexical Sample data sets. Among an en- semble of four different kinds of voted models, the KPCA-based model, along with the maximum en- tropy model, outperforms the boosting model and na¨ıve Bayes model. Interestingly, while the KPCA- based model typically achieves close or better ac- curacy than the maximum entropy model, neverthe- less a comparison of predicted classifications shows that it has a significantly different bias. This char- acteristic makes it an excellent voter, as confirmed by results showing that removing the KPCA-based model from the ensemble generally degrades per- formance. 1 Introduction Classifier combination has become a standard ar- chitecture for shared task evaluations in word sense disambiguation (WSD), named entity recognition, and similar problems that can naturally be cast as classification problems. Voting is the most com- mon method of combination, having proven to be remarkably effective yet simple. A key problem in improving the accuracy of such ensemble classification systems is to find new vot- ing models that (1) exhibit significantly different prediction biases from the models already voting, and yet (2) attain stand-alone classification accura- cies that are as good or better. When either of these conditions is not met, adding the new voting model typically degrades the accuracy of the ensemble in- stead of helping it. In this work, we investigate the potential of one promising new disambiguation model with respect 1The author would like to thank the Hong Kong Research Grants Council (RGC) for supporting this research in part through research grants RGC6083/99E, RGC6256/00E, and DAG03/04.EG09. to augmenting our existing ensemble combining a maximum entropy model, a boosting model, and a na¨ıve Bayes model—a combination representing some of the best stand-alone WSD models cur- rently known. The new WSD model, proposed by Wu et al. (2004), is a method for disambiguat- ing word senses that exploits a nonlinear Kernel Principal Component Analysis (KPCA) technique. That the KPCA-based model could potentially be a good candidate for a new voting model is sug- gested by Wu et al.’s empirical results showing that it yielded higher accuracies on Senseval-2 data sets than other models that included maximum entropy, na¨ıve Bayes, and SVM based models. In the following sections, we begin with a de- scription of the experimental setup, which utilizes a number of individual classifiers in a voting en- semble. We then describe the KPCA-based model to be added to the baseline ensemble. The accuracy results of the three submitted models are examined, and also the individual voting models are compared. Subsequently, we analyze the degree of difference in voting bias of the …

Content truncated to 500 words