SemEval-2007 Task 02: Evaluating Word Sense Induction and Discrimination Systems

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 7–12, Prague, June 2007. c⃝2007 Association for Computational Linguistics Semeval-2007 Task 02: Evaluating Word Sense Induction and Discrimination Systems Eneko Agirre IXA NLP Group Univ. of the Basque Country Donostia, Basque Country e.agirre@ehu.es Aitor Soroa IXA NLP Group Univ. of the Basque Country Donostia, Basque Country a.soroa@ehu.es Abstract The goal of this task is to allow for com- parison across sense-induction and discrim- ination systems, and also to compare these systems to other supervised and knowledge- based systems. In total there were 6 partic- ipating systems. We reused the SemEval- 2007 English lexical sample subtask of task 17, and set up both clustering-style unsuper- vised evaluation (using OntoNotes senses as gold-standard) and a supervised evaluation (using the part of the dataset for mapping). We provide a comparison to the results of the systems participating in the lexical sam- ple subtask of task 17. 1 Introduction Word Sense Disambiguation (WSD) is a key enabling-technology. Supervised WSD techniques are the best performing in public evaluations, but need large amounts of hand-tagging data. Exist- ing hand-annotated corpora like SemCor (Miller et al., 1993), which is annotated with WordNet senses (Fellbaum, 1998) allow for a small improve- ment over the simple most frequent sense heuristic, as attested in the all-words track of the last Sense- val competition (Snyder and Palmer, 2004). In the- ory, larger amounts of training data (SemCor has approx. 500M words) would improve the perfor- mance of supervised WSD, but no current project exists to provide such an expensive resource. An- other problem of the supervised approach is that the inventory and distribution of senses changes dra- matically from one domain to the other, requiring additional hand-tagging of corpora (Mart´ınez and Agirre, 2000; Koeling et al., 2005). Supervised WSD is based on the “fixed-list of senses” paradigm, where the senses for a target word are a closed list coming from a dictionary or lex- icon. Lexicographers and semanticists have long warned about the problems of such an approach, where senses are listed separately as discrete enti- ties, and have argued in favor of more complex rep- resentations, where, for instance, senses are dense regions in a continuum (Cruse, 2000). Unsupervised Word Sense Induction and Dis- crimination (WSID, also known as corpus-based un- supervised systems) has followed this line of think- ing, and tries to induce word senses directly from the corpus. Typical WSID systems involve cluster- ing techniques, which group together similar exam- ples. Given a set of induced clusters (which repre- sent word uses or senses1), each new occurrence of the target word will be compared to the clusters and the most similar cluster will be selected as its sense. One of the problems of unsupervised systems is that of managing to do a fair evaluation. Most of cur- rent unsupervised systems are evaluated in-house, with a brief comparison to a re-implementation of a former system, leading to a proliferation of unsuper- vised systems with little ground to compare …

Content truncated to 500 words