SemEval-2013 Task 11: Word Sense Induction and Disambiguation within an End-User Application

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 193–201, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics SemEval-2013 Task 11: Word Sense Induction & Disambiguation within an End-User Application Roberto Navigli and Daniele Vannella Dipartimento di Informatica Sapienza Universit`a di Roma Viale Regina Elena, 295 – 00161 Roma Italy {navigli,vannella}@di.uniroma1.it Abstract In this paper we describe our Semeval-2013 task on Word Sense Induction and Dis- ambiguation within an end-user application, namely Web search result clustering and diver- sification. Given a target query, induction and disambiguation systems are requested to clus- ter and diversify the search results returned by a search engine for that query. The task en- ables the end-to-end evaluation and compari- son of systems. 1 Introduction Word ambiguity is a pervasive issue in Natural Lan- guage Processing. Two main techniques in compu- tational lexical semantics, i.e., Word Sense Disam- biguation (WSD) and Word Sense Induction (WSI) address this issue from different perspectives: the former is aimed at assigning word senses from a pre- defined sense inventory to words in context, whereas the latter automatically identifies the meanings of a word of interest by clustering the contexts in which it occurs (see (Navigli, 2009; Navigli, 2012) for a survey). Unfortunately, the paradigms of both WSD and WSI suffer from significant issues which hamper their success in real-world applications. In fact, the performance of WSD systems depends heavily on which sense inventory is chosen. For instance, the most popular computational lexicon of English, i.e., WordNet (Fellbaum, 1998), provides fine-grained distinctions which make the disambiguation task quite difficult even for humans (Edmonds and Kil- garriff, 2002; Snyder and Palmer, 2004), although disagreements can be solved to some extent with graph-based methods (Navigli, 2008). On the other hand, although WSI overcomes this issue by allow- ing unrestrained sets of senses, its evaluation is par- ticularly arduous because there is no easy way of comparing and ranking different representations of senses. In fact, all the proposed measures in the lit- erature tend to favour specific cluster shapes (e.g., singletons or all-in-one clusters) of the senses pro- duced as output. Indeed, WSI evaluation is actually an instance of the more general and difficult problem of evaluating clustering algorithms. Nonetheless, many everyday tasks carried out by online users would benefit from intelligent systems able to address the lexical ambiguity issue effec- tively. A case in point is Web information retrieval, a task which is becoming increasingly difficult given the continuously growing pool of Web text of the most wildly disparate kinds. Recent work has ad- dressed this issue by proposing a general evaluation framework for injecting WSI into Web search result clustering and diversification (Navigli and Crisa- fulli, 2010; Di Marco and Navigli, 2013). In this task the search results returned by a search engine for an input query are grouped into clusters, and di- versified by providing a reranking which maximizes the meaning heterogeneity of the top ranking results. The …

Content truncated to 500 words