SemEval-2015 Task 17: Taxonomy Extraction Evaluation (TExEval)

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 902–910, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SemEval-2015 Task 17: Taxonomy Extraction Evaluation (TExEval) Georgeta Bordea, Paul Buitelaar Insight Centre for Data Analytics National University of Ireland, Galway name.surname@insight-centre.org Stefano Faralli, Roberto Navigli Dipartimento di Informatica Sapienza University of Rome Italy surname@di.uniroma1.it Abstract This paper describes the first shared task on Taxonomy Extraction Evaluation organ- ised as part of SemEval-2015. Participants were asked to find hypernym-hyponym re- lations between given terms. For each of the four selected target domains the partici- pants were provided with two lists of domain- specific terms: a WordNet collection of terms and a well-known terminology extracted from an online publicly available taxonomy. A total of 45 taxonomies submitted by 6 participating teams were evaluated using standard structural measures, the structural similarity with a gold standard taxonomy, and through manual qual- ity assessment of sampled novel relations. 1 Introduction SemEval-2015 Task 17 is concerned with the auto- matic extraction of hierarchical relations from text and subsequent taxonomy construction. A taxon- omy is a hierarchy of concepts that expresses parent- child or broader-narrower relationships. Because of their many applications in search, retrieval, website navigation, and records management, taxonomies are valuable resources for libraries, publishing com- panies, online databases, and e-commerce compa- nies. Taxonomies are most often manually created resources that are expensive to construct and main- tain, and therefore there is a need for automatic methods for taxonomy enrichment and construction. Recently, the task of taxonomy learning from text, also called taxonomy induction, has received an in- creased interest in the natural language processing community, as taxonomical information is a valu- able input to many semantically intensive tasks in- cluding inference, question answering (Harabagiu et al., 2003) and textual entailment (Geffet and Dagan, 2005). Taxonomy learning can be divided into three main subtasks: term extraction, relation discovery, and taxonomy construction. Term extraction is a rel- atively well-known task, hence we decided to ab- stract from this stage and provide a common ground for the next steps by making available the list of terms beforehand. Most approaches for relation dis- covery from text rely on lexico-syntactic patterns (Hearst, 1992; Kozareva et al., 2008), co-occurrence information (Sanderson and Croft, 1999), substring inclusion (Nevill-Manning et al., 1999), or exploit semantic relations provided in textual definitions (Navigli and Velardi, 2010). Any asymmetrical rela- tion that indicates subordination between two terms can be considered, but here the focus is mainly on hyponym-hypernym relations. Depending on the ap- proach selected, the task may or may not require large amounts of text to extract relations between terms, therefore no corpus is provided as part of the shared dataset. This stage usually produces a large number of noisy, inconsistent relations, that assign multiple parents to a node and that contain cycles, i.e., se- quences of vertices that start and end at the same vertex. Hence, the third stage of taxonomy learn- ing, taxonomy construction, focuses on the overall structure …

Content truncated to 500 words