SemEval-2010 Task 17: All-Words Word Sense Disambiguation on a Specific Domain

Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 75–80, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics SemEval-2010 Task 17: All-words Word Sense Disambiguation on a Specific Domain Eneko Agirre, Oier Lopez de Lacalle IXA NLP group UBC Donostia, Basque Country {e.agirre,oier.lopezdelacalle}@ehu.es Christiane Fellbaum Department of Computer Science Princeton University Princeton, USA fellbaum@princeton.edu Shu-Kai Hsieh Department of English National Taiwan Normal University Taipei, Taiwan shukai@ntnu.edu.tw Maurizio Tesconi IIT CNR Pisa, Italy maurizio.tesconi@iit.cnr.it Monica Monachini ILC CNR Pisa, Italy monica.monachini@ilc.cnr.it Piek Vossen, Roxanne Segers Faculteit der Letteren Vrije Universiteit Amsterdam Amsterdam, Netherlands p.vossen@let.vu.nl,roxane.segers@gmail.com Abstract Domain portability and adaptation of NLP components and Word Sense Disambigua- tion systems present new challenges. The difficulties found by supervised systems to adapt might change the way we assess the strengths and weaknesses of supervised and knowledge-based WSD systems. Un- fortunately, all existing evaluation datasets for specific domains are lexical-sample corpora. This task presented all-words datasets on the environment domain for WSD in four languages (Chinese, Dutch, English, Italian). 11 teams participated, with supervised and knowledge-based sys- tems, mainly in the English dataset. The results show that in all languages the par- ticipants where able to beat the most fre- quent sense heuristic as estimated from general corpora. The most successful ap- proaches used some sort of supervision in the form of hand-tagged examples from the domain. 1 Introduction Word Sense Disambiguation (WSD) competitions have focused on general domain texts, as attested in previous Senseval and SemEval competitions (Kilgarriff, 2001; Mihalcea et al., 2004; Snyder and Palmer, 2004; Pradhan et al., 2007). Spe- cific domains pose fresh challenges to WSD sys- tems: the context in which the senses occur might change, different domains involve different sense distributions and predominant senses, some words tend to occur in fewer senses in specific domains, the context of the senses might change, and new senses and terms might be involved. Both super- vised and knowledge-based systems are affected by these issues: while the first suffer from differ- ent context and sense priors, the later suffer from lack of coverage of domain-related words and in- formation. The main goal of this task is to provide a mul- tilingual testbed to evaluate WSD systems when faced with full-texts from a specific domain. All datasets and related information are publicly avail- able from the task websites1. This task was designed in the context of Ky- oto (Vossen et al., 2008)2, an Asian-European project that develops a community platform for modeling knowledge and finding facts across lan- guages and cultures. The platform operates as a Wiki system with an ontological support that so- cial communities can use to agree on the mean- ing of terms in specific domains of their interest. Kyoto focuses on the environmental domain be- cause it poses interesting challenges for informa- tion sharing, but the techniques and platforms are 1http://xmlgroup.iit.cnr.it/SemEval2010/ and http://semeval2.fbk.eu/ 2http://www.kyoto-project.eu/ 75 independent of the application domain. The paper is structured as follows. We first present the preparation of the data. Section …

Content truncated to 500 words