Combining Heterogeneous Classifiers for Word-Sense Disambiguation

Combining Heterogeneous Classifiers for Word-Sense Disambiguation H! Tolga Ilhan, Sepandar D. Kamvar, Dan Klein, Christopher D. Manning and Kristina Toutanova Computer Science Department Stanford University Stanford, CA 94305-9040, USA Abstract The Stanford-CS224N system is an ensemble of sim- ple classifiers. The first-tier systems are heteroge- neous, consisting primarily of naive-Bayes variants, but also including vector space, memory-based, and other classifier types. These simple classifiers are combined by a second-tier classifier, which variously uses majority voting, weighted voting, or a maxi- mum entropy model. Results from SENSEVAL-2 lex- ical sample tasks indicate that, while the individual classifiers perform at a level comparable to middle- scoring team's systems, the combination achieves high performance. In this paper, we discuss both our system and lessons learned from its behavior. 1 Introduction The problem of supervised word sense disam- biguation (wsD) has been approached using many different classification algorithms, includ- ing naive Bayes, decision trees, decision lists, and memory-based learners. While it is un- questionable that certain algorithms are better suited to the WSD problem than others (for a comparison, see Mooney ( 1996)), it seems to be the case that, given similar features as input, various algorithms do not behave dramatically differently. This was seen in the SENSEVAL-2 re- sults where a large fraction of the systems had scores clustered in a fairly narrow region. We began building our system with 23 su- pervised WSD systems, each submitted by a student taking the natural language processing course (CS224N) at Stanford University. Stu- dents were free to imple1pent whatever WSD This paper is based on work supported in part by the National Science Foundation under Grants IIS-0085896 and IIS-9982226, by an NSF Graduate Fellowship, and by the Research Collaboration between NTT Communi- cation Science Laboratories, Nippon Telegraph and Tele- phone Corporation and CSLI, Stanford University. 87 Combimnion Classitier First-tier Classifiers Figure 1: Organization of the system. method they chose. While most implemented variants of naive Bayes, some implemented a range of other methods, including n-gram mod- els, vector space models, and even memory- based learners. Although none of these systems alone would have produced more than middle- level performance on the SENSEVAL-2 task, we decided to investigate how they would behave in combination. In section 2, we discuss the first-tier classifiers in greater depth and describe our methods of combination. Section 3 discusses performance, analyzing what benefit was found from combi- nation, and when. We also discuss aspects of the component systems which substantially in- fluenced overall performance. 2 The System Figure 1 shows the high-level organization of our system. First, each of the 23 classifiers is run with 5-fold cross-validation on the train- ing data. Classifiers are ranked, for each word, based on their held-out accuracy. In any given run of the system, for some k, the top k clas- sifiers are kept, while lower-ranking classifiers are discarded. These remaining classifiers are combined by one of three methods. • Majority voting: The sense output by the most classifiers is chosen. Ties are broken in …

Content truncated to 500 words