SemEval-2016 Task 13: Taxonomy Extraction Evaluation (TExEval-2)

Proceedings of SemEval-2016, pages 1081–1091, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SemEval-2016 Task 13: Taxonomy Extraction Evaluation (TExEval-2) Georgeta Bordea*, Els Lefever**, Paul Buitelaar* *Insight Centre for Data Analytics National University of Ireland, Galway name.surname@insight-centre.org **LT3, Language and Translation Technology Team, Ghent University, Belgium name.surname@ugent.be Abstract This paper describes the second edition of the shared task on Taxonomy Extraction Evalua- tion organised as part of SemEval 2016. This task aims to extract hypernym-hyponym rela- tions between a given list of domain-specific terms and then to construct a domain taxon- omy based on them. TExEval-2 introduced a multilingual setting for this task, covering four different languages including English, Dutch, Italian and French from domains as diverse as environment, food and science. A total of 62 runs submitted by 5 different teams were evaluated using structural measures, by com- parison with gold standard taxonomies and by manual quality assessment of novel relations. 1 Introduction Taxonomies are useful tools for content organisa- tion, navigation, and retrieval, providing valuable in- put for semantically intensive tasks such as question answering (Harabagiu et al., 2003) and textual en- tailment (Geffet and Dagan, 2005). In general, a hi- erarchical relation is any asymmetrical relation that indicates subordination between two terms, but in this task we focus on hyponym-hypernym relations. Taxonomy learning from text is a challenging task that can be divided in several subtasks, including term extraction, hypernym identification and taxon- omy construction. Existing approaches for hyper- nym identification from text rely on lexico-syntactic patterns (Hearst, 1992; Lefever et al., 2014), cooc- currence information (Grefenstette, 2015), substring inclusion, or exploit semantic relations provided in textual definitions (Velardi et al., 2013). This stage usually produces a large number of noisy, incon- sistent relations, which assign multiple parents to a node and contain cycles. Hence, the third stage of taxonomy learning, taxonomy construction, focuses on the overall structure of the resulting graph and aims to organise terms in a hierarchical structure, more specifically a directed acyclic graph (Velardi et al., 2013; Kozareva and Hovy, 2010). More recently, the hypernym identification sub- task has attracted an increased interest from the distributional semantics community (Santus et al., 2014; Rei and Briscoe, 2014; Roller et al., 2014; Yu et al., 2015), as part of a wider effort to distinguish between different semantic relations which exist be- tween distributional similar words (Weeds et al., 2014; Levy et al., 2015). Although this is a promis- ing direction of research, that addresses some of the limitations of pattern-based approaches, including low coverage of domain-specific terms, most partic- ipants in this shared task opted for traditional ap- proaches for hypernym identification, with the ex- ception of one system (Pocostales, 2016). TexEval-2 is mainly concerned with automati- cally extracting hierarchical relations from text and subsequent taxonomy construction, therefore we make the assumption that a list of terms is readily available. This simplifies evaluation by providing a common ground for all the systems, but participants are allowed to add additional nodes, i.e. terms, …

Content truncated to 500 words