Performance metrics for word sense disambiguation

Performance Metrics for Word Sense Disambiguation Trevor Cohn Department of Computer Science and Software Engineering University of Melbourne, VIC 3010, Australia email: tacohn@cs.mu.oz.au fax: +61-3-9348-1184 Abstract This paper presents the area under the Receiver Operating Characteristics (ROC) curve as an alternative metric for evaluating word sense disambigua- tion performance. The current met- rics – accuracy, precision and recall – while suitable for two-way classifica- tion, are shown to be inadequate when disambiguating between three or more senses. Specifically, these measures do not facilitate comparison with base- line performance nor are they sensitive to non-uniform misclassification costs. Both of these issues can be addressed using ROC analysis. 1 Introduction Word sense disambiguation (WSD) is one of the large open problems in the field of natural lan- guage processing, and in recent years has at- tracted considerable research interest (Ide and Veronis, 1998). The increasing availability of large corpora along with electronic sense invento- ries (such as WordNet; Fellbaum (1998)) has per- mitted the application of a raft of machine learn- ing techniques to the task and provided an em- pirical means of performance evaluation. Until recently, most performance evaluation was con- ducted on disparate data sets, with only the line and interest corpora being used in a significant number of studies (Leacock et al., 1993; Bruce and Wiebe, 1994). SENSEVAL, a global eval- uation performed in 1998 (Kilgarriff, 1998) and again in 2001 (Edmonds and Cotton, 2001), pro- vided a common set of disambiguation tasks and performance evaluation criteria, allowing an ob- jective comparison between competing methods. These workshops included the tasks of disam- biguating all words in a given text (the all-words task), and disambiguating each occurrence of a given word when it appears with a short context of a few surrounding sentences (the lexical sample task). Performance in the two tasks was measured in terms of precision and recall. Precision was de- fined as the proportion of classified instances that were correctly classified, and recall as the propor- tion of instances classified correctly – these allow for the possibility of an algorithm choosing not to classify a given instance. This evaluation criterion is insensitive to both the type of misclassification (is the predicted sense more closely related to the correct sense than other possible senses?) and the confidence with which the classifier has made the prediction (is the correct sense allocated a high probability despite not being given the highest value by the classifier?). These problems led Resnik and Yarowsky (1999) to suggest an evaluation metric to provide partial credit for incorrectly classified instances. They penalise probability mass assigned to incor- rect senses weighted by what they term the com- municative/semantic distance between the that predicted sense and the correct sense. Using such measures, systems that confuse homographs would be penalised most heavily, while those that confuse fine-grained senses would only attract a minor penalty. The score assigned to a particu- lar algorithm is highly reliant on the distances be- tween senses; altering the relative penalties may well promote …

Content truncated to 500 words