A Perspective on Word Sense Disambiguation Methods and Their Evaluation

A Perspective on Word Sense Disambiguation Methods and Their Evaluation Philip Resnik Dept. of Linguistics/UMIACS University of Maryland College Park, MD 20742 resnik@umiacs, umd. edu David Yarowsky Dept. of Computer Science/CLSP Johns Hopkins University Baltimore, MD 21218 yarowsky@cs, jhu. edu Abstract In this position paper, we make several observations about the state of the art in automatic word sense disambiguation. Mo- tivated by these observations, we offer sev- eral specific proposals to the community re- garding improved evaluation criteria, com- mon training and testing resources, and the definition of sense inventories. 1 Introduction Word sense disambiguation (WSD) is perhaps the great open problem at the lexical level of natural language processing. If one requires part-of-speech tagging for some task, it is now possible to obtain high performance off the shelf; if one needs morpho- logical analysis, software and lexical data are not too hard to find. In both cases, performance of state- of-the-art systems is respectable, if not perfect, and the fundamentals of the dominant approaches (noisy channel models for tagging, two-level morphology) are by now well understood. For word sense disam- biguation, we have a far longer way to go. 2 Observations Observation 1. Evaluation of word sense dis- ambiguation systems is not yet standardized. Evaluation of many natural language processing tasks including part-of-speech tagging and parsing has become fairly standardized, with most reported studies using common training and testing resources such as the Brown Corpus and Penn Treebank. Per- formance measures include a fairly well recognized suite of metrics including crossing brackets and pre- cision/recall of non-terminal label placement. Sev- eral researchers (including Charniak, Collins and Magerman) have facilitated contrastive evaluation of their parsers by even training and testing on identi- cal segments of the Treebank. Government funding agencies have accelerated this process, and even the task of anaphora resolution has achieved an evalua- tion standard under the MUC-6 program. In contrast, most previous work in word sense dis- ambiguation has tended to use different sets of poly- semous words, different corpora and different eval- uation metrics. Some clusters of studies have used common test suites, most notably the 2094-word Hne data of Leacock et al. (1993), shared by Lehman (1994) and Mooney (1996) and evaluated on the sys- tem of Gale, Church and Yarowsky (1992). Also, re- searchers have tended to keep their evaluation data and procedures somewhat standard across their own studies for internally consistent comparison. Never- theless, there are nearly as many test suites as there are researchers in this field. Observation 2. The potential for WSD varies by task. As Wilks and Stevenson (1996) empha- size, disambiguating word senses is not an end in itself, but rather an intermediate capability that is believed -- but not yet proven -- to improve natu- ral language applications. It would appear, however, that different major applications of language differ in their potential to make use of successful word sense information. In information retrieval, even perfect word sense information may be of only limited util- ity, largely owing …

Content truncated to 500 words