OE: WSD Using Optimal Ensembling (OE) Method

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 257–260, Prague, June 2007. c⃝2007 Association for Computational Linguistics OE: WSD Using Optimal Ensembling (OE) Method Harri M. T. Saarikoski Helsinki University Language Technology PhD Programme F-00014 Helsinki, Finland harri.saarikoski@helsinki.fi Abstract Optimal ensembling (OE) is a word sense disambiguation (WSD) method using word-specific training factors (average pos- itive vs negative training per sense, posex and negex) to predict best system (classifier algorithm / applicable feature set) for given target word. Our official entry (OE1) in Senseval-4 Task 17 (coarse-grained En- glish lexical sample task) contained many design flaws and thus failed to show the whole potential of the method, finishing -4.9% behind top system (+0.5 gain over best base system). A fixed system (OE2) finished only -3.4% behind (+2.0% net gain). All our systems were 'closed', i.e. used the official training data only (average 56 training examples per each sense). We also show that the official evaluation mea- sure tends to favor systems that do well with high-trained words. 1 Introduction Optimal ensembling is a novel method for combining WSD systems and obtaining higher classification accuracy (presented more fully in Saarikoski et al. 2007). The essential difference from other ensembling methods (such as various types of voting ensembles and cross-validation based best machine selection) is that best machine is predicted using factors calculated from words (e.g. number of senses) and their training data (e.g. number of training examples per sense). The method is loosely based on findings of system performance differences in both WSD (different machines by Yarowsky et al., 2002 and different feature sets by Mihalcea, 2002) and other classification tasks such as text categorization (Forman et al., 2004, Bay et al., 2002). 2 Method We first describe in detail the two selection routines in OE as deployed in this experiment. 2.1 Machine (Mach) Selection We selected support vector machine (SVM) (Vapnik, 1995) and Naive Bayes (NB) (John et al. 1995) as classifiers for our base systems to be optimally ensembled. This was mainly because of their attested strength at earlier Senseval evaluations (Edmonds et al. 2002, Mihalcea et al. 2004) and mutual complementarity discovered by us (Saarikoski et al., 2007). Original batch of candidate machines that we tested for OE using Senseval-2 dataset included the following classifiers: Decision Stump, Decision Tree with various values of confidence (c) parameter 0.05, 0.15, 0.25 and instance-based classifier with k values ranging from 1..15 at intervals of two 1. After cross-validation runs against current dataset (see below), however, SVM and NB proved again to be overall strongest regardless of training input, so we built OE around those two classifiers. 2.2 Feature Set (Fset) Selection We extracted three contextual feature sets from training data for all words to train the machines: 1- grams (1g) and sequential 2-grams both from whole instance (2g) as well as part-of-speech tags from local 1-word window around and including target word (pos3). We also used three 'multifsets' (1g-2g, 1g-pos3, 2g-pos3). 1We used Weka implementations (J48, Ibk, SMO, …

Content truncated to 500 words