Discriminator at SemEval-2018 Task 10: Minimally Supervised Discrimination

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 1008–1012 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics Discriminator at SemEval-2018 Task 10: Minimally Supervised Discrimination Artur Kulmizev CLCG University of Groningen a.kulmizev@student.rug.nl Mostafa Abdou CLCG University of Groningen m.abdou@student.rug.nl Vinit Ravishankar Institute of Formal and Applied Linguistics Charles University in Prague vinit.ravishankar@gmail.com Malvina Nissim CLCG University of Groningen m.nissim@rug.nl Abstract We participated to the SemEval-2018 shared task on capturing discriminative attributes (Task 10) with a simple system that ranked 8th amongst the 26 teams that took part in the evaluation. Our final score was 0.67, which is competitive with the winning score of 0.75, particularly given that our system is a mini- mally supervised system that requires no train- ing and minimal parameter optimisation. In addition to describing the submitted system, and discussing the implications of the relative success of such a system on this task, we also report on other, more complex models we ex- perimented with. 1 Introduction and Background Traditional evaluation tasks for semantic mod- els have aimed to evaluate semantic relatedness (Bruni et al., 2014; Agirre et al., 2009) or similar- ity (Hill et al., 2014). More recently, analogy tasks (Mikolov et al., 2013; Gladkova et al., 2016; Ab- dou et al., 2018) have emerged in order to assess a model’s ability to correctly answer questions in the form of “a is to b as c is to ?” using vector arithmetic. However, these approaches are not suf- ficient in evaluating the semantic competence of any given model: there are numerous flaws with similarity and analogy-based evaluation, the most pressing being the lack of correlation with down- stream performance in real-world tasks (Schnabel et al., 2015). Furthermore, though analogy ques- tions can assess how well certain semantic rela- tions are modeled (country : capital, country: lan- guage), this is arguably more of a measure of con- text co-occurrence than it is a testament to any se- mantic understanding. Capturing Discriminative Attributes is a novel semantic evaluation task which aims to assess the extent to which semantic models can capture se- mantic differences between words. Particularly, the task is concerned with identifying how well a given model represents attributes that discrimi- nate between two related semantic concepts. For instance, given the terms steak and salad, meat would serve as a discriminative attribute, drawing a distinction between the former and latter as a quality that the two do not have in common. 2 Data The data provided for this task was split between training and validation sections, with 17,500 and 2,721 samples comprising the former and latter, respectively. Every sample was composed of three terms (pivot, comparison, and feature) and their corresponding label ∈ {0, 1}. A sample was deemed as discriminative (label 1) if the feature served as an attribute that distinguished the pivot from the comparison. Otherwise, the sample was labeled as non-discriminative (0). Examples from the data are shown in Table 2. Please note that di- rectionality is meaningful, …

Content truncated to 500 words