SemEval-2012 Task 2: Measuring Degrees of Relational Similarity

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 356–364, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics SemEval-2012 Task 2: Measuring Degrees of Relational Similarity David A. Jurgens Department of Computer Science University of California, Los Angeles jurgens@cs.ucla.edu Saif M. Mohammad Emerging Technologies National Research Council Canada saif.mohammad@nrc-cnrc.gc.ca Peter D. Turney Emerging Technologies National Research Council Canada peter.turney@nrc-cnrc.gc.ca Keith J. Holyoak Department of Psychology University of California, Los Angeles holyoak@lifesci.ucla.edu Abstract Up to now, work on semantic relations has fo- cused on relation classification: recognizing whether a given instance (a word pair such as virus:flu) belongs to a specific relation class (such as CAUSE:EFFECT). However, instances of a single relation class may still have signif- icant variability in how characteristic they are of that class. We present a new SemEval task based on identifying the degree of prototypi- cality for instances within a given class. As a part of the task, we have assembled the first dataset of graded relational similarity ratings across 79 relation categories. Three teams submitted six systems, which were evaluated using two methods. 1 Introduction Relational similarity measures the degree of corre- spondence between two relations, where instance pairs that have high relational similarity are said to be analogous, i.e., to express the same relation (Tur- ney, 2006). However, a class of analogous relations may still have significant variability in the degree of relational similarity of its members. Consider the four word pairs dog:bark, cat:meow, floor:squeak, and car:honk. We could say that these four X:Y pairs are all instances of the semantic relation EN- TITY:SOUND; that is, X is an entity that character- istically makes the sound Y . Within a class of anal- ogous pairs, certain pairs are more characteristic of the relation. For example, many would agree that dog:bark and cat:meow are better prototypes of the ENTITY:SOUND relation than floor:squeak. Our task requires automatic systems to quantify the degree of prototypicality of a target pair by measuring the re- lational similarity between it and pairs that are given as defining examples of a particular relation. So far, most work in semantic relations has fo- cused on differences between relation categories for classifying new relation instances. Past SemEval tasks that use relations have focused largely on dis- crete classification (Girju et al., 2007; Hendrickx et al., 2010) and paraphrasing the relations connecting noun compounds with a verb (Butnariu et al., 2010), which is also a form of discrete classification due to the lack of continuous degrees. However, there is some loss of information in any discrete classifica- tion of semantic relations. Furthermore, while some discrete classifiers provide a degree of confidence or probability for a relation classification, there is no a priori reason that such values would correspond to human prototypicality judgments. Our proposed task is distinct from these past tasks in that we fo- cus on measuring the degree of relational similarity.1 A graded measure of the degree of relational simi- larity would tell us that dog:bark is more similar …

Content truncated to 500 words