SemEval-2016 Task 14: Semantic Taxonomy Enrichment

Proceedings of SemEval-2016, pages 1092–1102, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SemEval-2016 Task 14: Semantic Taxonomy Enrichment David Jurgens Stanford University jurgens@cs.stanford.edu Mohammad Taher Pilehvar University of Cambridge mp792@cam.ac.uk Abstract Manually constructed taxonomies provide a crucial resource for many NLP technologies, yet these resources are often limited in their lexical coverage due to their construction pro- cedure. While multiple approaches have been proposed to enrich such taxonomies with new concepts, these techniques are typically eval- uated by measuring the accuracy at identify- ing relationships between words, e.g., that a dog is a canine, rather relationships between specific concepts. Task 14 provides an evalu- ation framework for automatic taxonomy en- richment techniques by measuring the place- ment of a new concept into an existing tax- onomy: Given a new word and its definition, systems were asked to attach or merge the con- cept into an existing WordNet concept. Five teams submitted 13 systems to the task, all of which were able to improve over the random baseline system. However, only one partici- pating system outperformed the second, more- competitive baseline that attaches a new term to the first word in its gloss with the appropri- ate part of speech, which indicates that tech- niques must be adapted to exploit the structure of glosses. 1 Introduction Semantic networks and ontologies are key resources in Natural Language Processing. Of these re- sources, WordNet (Fellbaum, 1998), the de facto standard lexical database of English, has remained in widespread use over the past two decades, with a broad range of applications such as Word Sense Disambiguation (Navigli, 2009), Query expansion and Information Retrieval (Varelas et al., 2005; Fang, 2008), sentiment analysis (Esuli and Sebas- tiani, 2006), and semantic similarity measurement (Budanitsky and Hirst, 2006a; Pilehvar et al., 2013). The performances of these WordNet-based tech- niques are directly affected by the lexical cover- age of WordNet’s vocabulary, especially if applied to specific domains and social media texts. How- ever, the manual maintenance of WordNet is an ex- pensive endeavour which requires significant effort and time. As a result, WordNet is not updated fre- quently and omits many lemmas and senses, such as those from domain specific lexicons (e.g., DNA replication, regular expression, and long shot), cre- ative slang usages (e.g., homewrecker), or those for technology or entities that came into recent exis- tence (e.g., selfie, mp3). Hence, a variety of techniques have tried to tackle the coverage limitation of WordNet, often by draw- ing new word senses from other domain-specific or collaboratively-constructed dictionaries and adding the new word senses to the WordNet hierarchy (Poprat et al., 2008; Snow et al., 2006; Toral et al., 2008; Yamada et al., 2011; Jurgens and Pile- hvar, 2015). However, these approaches have usu- ally been tested on relatively small datasets, of- ten testing for word-level relationships without pre- cisely measuring integration accuracy at the concept level. Similarly, other techniques have been pro- posed for automatically discovering novel senses of words (Lau et al., 2012); however, …

Content truncated to 500 words