Semi-supervised training of a Kernel PCA-Based Model for Word Sense Disambiguation

Semi-Supervised Training of a Kernel PCA-Based Model for Word Sense Disambiguation Weifeng SU Marine CARPUAT Dekai WU1 weifeng@cs.ust.hk marine@cs.ust.hk dekai@cs.ust.hk Human Language Technology Center HKUST Department of Computer Science University of Science and Technology, Clear Water Bay, Hong Kong Abstract In this paper, we introduce a new semi-supervised learning model for word sense disambiguation based on Kernel Prin- cipal Component Analysis (KPCA), with experiments showing that it can further improve accuracy over supervised KPCA models that have achieved WSD accuracy superior to the best published individual models. Although empirical results with supervised KPCA models demonstrate significantly better ac- curacy compared to the state-of-the-art achieved by either na¨ıve Bayes or maximum entropy models on Senseval-2 data, we identify specific sparse data conditions under which supervised KPCA models deteriorate to essentially a most-frequent-sense predictor. We discuss the potential of KPCA for leveraging unannotated data for partially-unsupervised training to address these issues, leading to a composite model that combines both the supervised and semi-supervised models. 1 Introduction Wu et al. (2004) propose an efficient and accurate new supervised learning model for word sense disambigua- tion (WSD), that exploits a nonlinear Kernel Principal Component Analysis (KPCA) technique to make pre- dictions implicitly based on generalizations over feature combinations. Experiments performed on the Senseval- 2 English lexical sample data show that KPCA-based word sense disambiguation method is capable of outper- forming other widely used WSD models including na¨ıve Bayes, maximum entropy, and SVM models. Despite the excellent performance of the supervised KPCA-based WSD model on average, though, our fur- ther error analysis investigations have suggested certain limitations. In particular, the supervised KPCA-based model often appears to perform poorly when it encoun- ters target words whose contexts are highly dissimilar to those of any previously seen instances in the train- ing set. Empirically, the supervised KPCA-based model nearly always disambiguates target words of this kind to the most frequent sense. As a result, for this partic- ular subset of test instances, the precision achieved by the KPCA-based model is essentially no higher than the precision achieved by the most-frequent-sense baseline model (which simply always selects the most frequent sense for the target word). The work reported in this pa- per stems from a hypothesis that the most-frequent-sense 1The author would like to thank the Hong Kong Research Grants Council (RGC) for supporting this research in part through grants RGC6083/99E, RGC6256/00E, and DAG03/04.EG09. strategy can be bettered for this category of errors. This is a case of data sparseness, so the observation should not be very surprising. Such behavior is to be ex- pected from classifiers in general, and not just the KPCA- based model. Put another way, even though KPCA is able to generalize over combinations of dependent fea- tures, there must be a sufficient number of training in- stances from which to generalize. The nature of KPCA, however, suggests a strategy that is not applicable to many of the other conventional WSD models. We propose a model in this paper that takes ad- vantage of …

Content truncated to 500 words