SemEval-2010 Task 14: Word Sense Induction &Disambiguation

Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 63–68, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics SemEval-2010 Task 14: Word Sense Induction & Disambiguation Suresh Manandhar Department of Computer Science University of York, UK Ioannis P. Klapaftis Department of Computer Science University of York, UK Dmitriy Dligach Department of Computer Science University of Colorado, USA Sameer S. Pradhan BBN Technologies Cambridge, USA Abstract This paper presents the description and evaluation framework of SemEval-2010 Word Sense Induction & Disambiguation task, as well as the evaluation results of 26 participating systems. In this task, partici- pants were required to induce the senses of 100 target words using a training set, and then disambiguate unseen instances of the same words using the induced senses. Sys- tems’ answers were evaluated in: (1) an unsupervised manner by using two clus- tering evaluation measures, and (2) a su- pervised manner in a WSD task. 1 Introduction Word senses are more beneficial than simple word forms for a variety of tasks including Information Retrieval, Machine Translation and others (Pantel and Lin, 2002). However, word senses are usually represented as a fixed-list of definitions of a manu- ally constructed lexical database. Several deficien- cies are caused by this representation, e.g. lexical databases miss main domain-specific senses (Pan- tel and Lin, 2002), they often contain general defi- nitions and suffer from the lack of explicit seman- tic or contextual links between concepts (Agirre et al., 2001). More importantly, the definitions of hand-crafted lexical databases often do not reflect the exact meaning of a target word in a given con- text (V´eronis, 2004). Unsupervised Word Sense Induction (WSI) aims to overcome these limitations of hand- constructed lexicons by learning the senses of a target word directly from text without relying on any hand-crafted resources. The primary aim of SemEval-2010 WSI task is to allow comparison of unsupervised word sense induction and disam- biguation systems. The target word dataset consists of 100 words, 50 nouns and 50 verbs. For each target word, par- ticipants were provided with a training set in or- der to learn the senses of that word. In the next step, participating systems were asked to disam- biguate unseen instances of the same words using their learned senses. The answers of the systems were then sent to organisers for evaluation. 2 Task description Figure 1 provides an overview of the task. As can be observed, the task consisted of three separate phases. In the first phase, train- ing phase, participating systems were provided with a training dataset that consisted of a set of target word (noun/verb) instances (sen- tences/paragraphs). Participants were then asked to use this training dataset to induce the senses of the target word. No other resources were al- lowed with the exception of NLP components for morphology and syntax. In the second phase, testing phase, participating systems were pro- vided with a testing dataset that consisted of a set of target word (noun/verb) instances (sen- tences/paragraphs). Participants were …

Content truncated to 500 words