SemEval-2018 Task 9: Hypernym Discovery

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 712–724 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics SemEval-2018 Task 9: Hypernym Discovery Jose Camacho-Collados♣ Claudio Delli Bovi♥ Luis Espinosa-Anke♣ Sergio Oramas♦ Tommaso Pasini♥ Enrico Santus♥ Vered Shwartz♠ Roberto Navigli♥ Horacio Saggion♦ ♣School of Computer Science and Informatics, Cardiff University, United Kingdom ♥ Computer Science Department, Sapienza University of Rome, Italy ♦ Pompeu Fabra University, Barcelona, Spain ♥ MIT, United States ♠ Bar-Ilan University, Ramat Gan, Israel ♣{camachocolladosj,espinosa-ankel}@cardiff.ac.uk, ♥{dellibovi,pasini,navigli}@di.uniroma1.it, ♦{name.surname}@upf.edu, ♥esantus@mit.edu, ♠vered1986@gmail.com Abstract This paper describes the SemEval 2018 Shared Task on Hypernym Discovery. We put for- ward this task as a complementary benchmark for modeling hypernymy, a problem which has traditionally been cast as a binary classifica- tion task, taking a pair of candidate words as input. Instead, our reformulated task is de- fined as follows: given an input term, retrieve (or discover) its suitable hypernyms from a tar- get corpus. We proposed five different sub- tasks covering three languages (English, Span- ish, and Italian), and two specific domains of knowledge in English (Medical and Music). Participants were allowed to compete in any or all of the subtasks. Overall, a total of 11 teams participated, with a total of 39 different sys- tems submitted through all subtasks. Data, re- sults and further information about the task can be found at https://competitions. codalab.org/competitions/17119. 1 Introduction Hypernymy, i.e. the capability to relate generic terms or classes to their specific instances, lies at the core of human cognition. It is not surpris- ing, therefore, that identifying hypernymic (is-a) relations has been pursued in NLP for more than two decades (Shwartz et al., 2016): indeed, suc- cessfully identifying this lexical relation substan- tially improves Question Answering applications (Prager et al., 2008; Yahya et al., 2013), Textual Entailment and Semantic Search systems (Hoffart et al., 2014; Roller et al., 2014; Roller and Erk, 2016). In addition, hypernymic relations are the backbone of almost every ontology, semantic net- work and taxonomy (Yu et al., 2015), which are in turn useful resources for downstream tasks such as web retrieval, website navigation or records man- agement (Bordea et al., 2015). Generally, evaluation benchmarks for modeling hypernymy have been designed such that in most cases they are reduced to binary classification (Ba- roni and Lenci, 2011; Snow et al., 2004; Boleda et al., 2017; Vyas and Carpuat, 2017), where a system has to decide whether a hypernymic rela- tion holds between a given candidate pair of terms. Criticisms to this experimental setting point out that supervised systems tend to benefit from the inherent modeling of the datasets in the hyper- nym detection task, leading to lexical memoriza- tion phenomena (Levy et al., 2015; Santus et al., 2016a; Shwartz et al., 2017). In this respect, re- cent work has attempted to alleviate this issue by including a graded scale for evaluating the degree of hypernymy on a given pair (Vuli´c et al., 2017). Crucially, Espinosa-Anke et al. (2016) proposed to frame the problem as …

Content truncated to 500 words