A Methodology for Word Sense Disambiguation at 90% based on large-scale CrowdSourcing

Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics (*SEM 2015), pages 61–70, Denver, Colorado, June 4–5, 2015. A Methodology for Word Sense Disambiguation at 90% based on large-scale CrowdSourcing Oier Lopez de Lacalle University of the Basque Country oier.lopezdelacalle@ehu.eus Eneko Agirre University of the Basque Country e.agirre@ehu.eus Abstract Word Sense Disambiguation has been stuck for many years. In this paper we explore the use of large-scale crowdsourcing to cluster senses that are often confused by non-expert annotators. We show that we can increase performance at will: our in-domain experi- ment involving 45 highly polysemous nouns, verbs and adjective (9.8 senses on average), yields an average accuracy of 92.6 using a su- pervised classifier for an average polysemy of 6.1. Our proposal has the advantage of being cost-effective and being able to produce differ- ent levels of granularity. Our analysis shows that the error reduction with respect to fine- grained senses is higher, and manual inspec- tion show that the clusters are sensible when compared to those of OntoNotes and WordNet Supersenses. 1 Introduction Word sense ambiguity is a major hurdle for accu- rate information extraction, summarization and ma- chine translation. The utility of Word Sense Dis- ambiguation (WSD) depends on the accuracy and on how useful the sense distinctions are. The first issue is quantitative, as it can be measured using a WSD system on certain dataset. The second ex- amines whether the sense distinctions are appropri- ate, which varies from application to application. Although usefulness can be explored in a down- stream application (Agirre et al., 2008), it is usually assessed subjectively, discussing the quality of the sense distinctions (Palmer et al., 2007). Both issues (performance and usefulness) are linked to the gran- ularity of the sense inventory, and conflict with each other: finer granularity might produce more useful distinctions but the accuracy would be worse, and vice-versa. WordNet (Fellbaum, 1998) is the most widely used resource to build word sense disambiguation tools and word sense annotated corpora, including recent large efforts (Passonneau et al., 2012), but its fine-grainedness has been mentioned to be a prob- lem (Hovy et al., 2006; Palmer et al., 2007). We think that a desiderata for a sense inventory would be that it provides useful sense distinctions and useful performance across a large range of appli- cations. We would also add that it should be tightly integrated with WordNet, given its prevalence on NLP applications, and we thus focus on sense in- ventories which are mapped to WordNet. In order to asses usefulness, we need specific measures. Downstream application is difficult, and unfeasible for new proposals, as a full-fledged sense inventories and associated annotations are neces- sary. We can instead estimate usefulness of pro- posed sense inventories using several proxy mea- sures: • High polysemy. Note that polysemy alone could be misleading, as a word with many senses might be skewed to a single sense: 99% of occurrences could belong to a single sense, while the rest are only seen once. …

Content truncated to 500 words