A Kernel PCA Method for Superior Word Sense Disambiguation

A Kernel PCA Method for Superior Word Sense Disambiguation Dekai WU1 Weifeng SU Marine CARPUAT dekai@cs.ust.hk weifeng@cs.ust.hk marine@cs.ust.hk Human Language Technology Center HKUST Department of Computer Science University of Science and Technology Clear Water Bay, Hong Kong Abstract We introduce a new method for disambiguating word senses that exploits a nonlinear Kernel Prin- cipal Component Analysis (KPCA) technique to achieve accuracy superior to the best published indi- vidual models. We present empirical results demon- strating significantly better accuracy compared to the state-of-the-art achieved by either na¨ıve Bayes or maximum entropy models, on Senseval-2 data. We also contrast against another type of kernel method, the support vector machine (SVM) model, and show that our KPCA-based model outperforms the SVM-based model. It is hoped that these highly encouraging first results on KPCA for natural lan- guage processing tasks will inspire further develop- ment of these directions. 1 Introduction Achieving higher precision in supervised word sense disambiguation (WSD) tasks without resort- ing to ad hoc voting or similar ensemble techniques has become somewhat daunting in recent years, given the challenging benchmarks set by na¨ıve Bayes models (e.g., Mooney (1996), Chodorow et al. (1999), Pedersen (2001), Yarowsky and Flo- rian (2002)) as well as maximum entropy models (e.g., Dang and Palmer (2002), Klein and Man- ning (2002)). A good foundation for comparative studies has been established by the Senseval data and evaluations; of particular relevance here are the lexical sample tasks from Senseval-1 (Kilgarriff and Rosenzweig, 1999) and Senseval-2 (Kilgarriff, 2001). We therefore chose this problem to introduce an efficient and accurate new word sense disam- biguation approach that exploits a nonlinear Kernel PCA technique to make predictions implicitly based on generalizations over feature combinations. The 1The author would like to thank the Hong Kong Re- search Grants Council (RGC) for supporting this research in part through grants RGC6083/99E, RGC6256/00E, and DAG03/04.EG09. technique is applicable whenever vector represen- tations of a disambiguation task can be generated; thus many properties of our technique can be ex- pected to be highly attractive from the standpoint of natural language processing in general. In the following sections, we first analyze the po- tential of nonlinear principal components with re- spect to the task of disambiguating word senses. Based on this, we describe a full model for WSD built on KPCA. We then discuss experimental re- sults confirming that this model outperforms state- of-the-art published models for Senseval-related lexical sample tasks as represented by (1) na¨ıve Bayes models, as well as (2) maximum entropy models. We then consider whether other kernel methods—in particular, the popular SVM model— are equally competitive, and discover experimen- tally that KPCA achieves higher accuracy than the SVM model. 2 Nonlinear principal components and WSD The Kernel Principal Component Analysis tech- nique, or KPCA, is a nonlinear kernel method for extraction of nonlinear principal components from vector sets in which, conceptually, the n- dimensional input vectors are nonlinearly mapped from their original space Rn to a high-dimensional feature space F where linear PCA is performed, …

Content truncated to 500 words