SemEval-2018 Task 10: Capturing Discriminative Attributes

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 732–740 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics SemEval-2018 Task 10: Capturing Discriminative Attributes Alicia Krebs Textkernel krebs@ textkernel.nl Alessandro Lenci University of Pisa alessandro.lenci@ unipi.it Denis Paperno Loria (UMR 7503), CNRS denis.paperno@ loria.fr Abstract This paper describes the SemEval 2018 Task 10 on Capturing Discriminative Attributes. Participants were asked to identify whether an attribute could help discriminate between two concepts. For example, a successful system should determine that urine is a discriminating feature in the word pair kidney,bone. The aim of the task is to better evaluate the capabili- ties of state of the art semantic models, beyond pure semantic similarity. The task attracted submissions from 21 teams, and the best sys- tem achieved a 0.75 F1 score. 1 Introduction State of the art semantic models do an excellent job at detecting semantic similarity, a traditional semantic task; for example, they can tell us that cappuccino, espresso and americano are similar to each other. It is obvious, however, that no model can claim to capture semantic competence if it does not, in addition to similarity, predict seman- tic differences between words. If one can tell that americano is similar to cappuccino and espresso but cannot tell the difference between them, one only has a very approximate idea of the meaning of these words. As a step beyond similarity, one should at the very least recognize that americano is bigger than espresso, and that capuccino contains milk foam. In this spirit, we present Semeval 2018 Task 10 (Capturing Discriminative Attributes) as a new challenge for lexical semantic models. 1.1 Task description A semantic model that has only been evaluated on similarity detection may very well fail to be of practical use for specific applications. For ex- ample, word sense disambiguation could benefit greatly from representations that can model com- plex semantic relations. This means that the eval- uation of word representation models should not only be centered on semantic similarity and relat- edness, and should include different, complemen- tary tasks. To fill this gap, we proposed a novel task of semantic difference detection as Task 10 of the SemEval 2018 workshop. The goal of the sys- tems in this case was to predict whether a word is a discriminative attribute between two other words. For example, given the words apple and banana, is the word red a discriminative attribute? Semantic difference is a ternary relation be- tween two concepts (apple, banana) and a dis- criminative attribute (red) that characterizes the first concept but not the other. By its nature, se- mantic difference detection is a binary classifica- tion task: given a triple apple,banana,red, the task is to determine whether it exemplifies a semantic difference or not. In practice, when preparing the task, we started out with defining potential discriminative attributes as semantic features in the sense of (McRae et al., 2005): properties that people tend to think are important for a given concept. McRae et …

Content truncated to 500 words