UMD at SemEval-2018 Task 10: Can Word Embeddings Capture Discriminative Attributes?

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 1022–1026 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics UMD at SemEval-2018 Task 10: Can Word Embeddings Capture Discriminative Attributes? Alexander Zhang and Marine Carpuat Department of Computer Science University of Maryland College Park, MD 20742, USA alexz@umd.edu, marine@cs.umd.edu Abstract We describe the University of Mary- land’s submission to SemEval-018 Task 10, “Capturing Discriminative Attributes”: given word triples (w1, w2, d), the goal is to determine whether d is a discriminat- ing attribute belonging to w1 but not w2. Our study aims to determine whether word embeddings can address this challenging task. Our submission casts this problem as supervised binary classification using only word embedding features. Using a gaus- sian SVM model trained only on valida- tion data results in an F-score of 60%. We also show that cosine similarity features are more effective, both in unsupervised systems (F-score of 65%) and supervised systems (F-score of 67%). 1 Introduction SemEval-2018 Task 10 (Krebs et al., 2018) offers an opportunity to evaluate word embeddings on a challenging lexical semantics problem. Much prior work on word embeddings has focused on the well-established task of detecting seman- tic similarity (Mikolov et al., 2013a; Pennington et al., 2014; Baroni et al., 2014; Upadhyay et al., 2016). However, semantic similarity tasks alone cannot fully characterize the differences in mean- ing between words. For example, we would ex- pect the word car to have high semantic similar- ity with truck and with vehicle in distributional vector spaces, while the relation between car and truck differs from the relation between car and vehicle. In addition, popular datasets for simi- larity tasks are small, and similarity annotations are subjective with low inter-annotator agreement (Krebs and Paperno, 2016). Task 10 focuses instead on determining seman- tic difference: given a word triple (w1, w2, d), the task consists in predicting whether d is a discrim- inating attribute applicable to w1, but not to w2. For instance, (w1 =apple, w2 =banana, d =red) is a positive example as red is a typical attribute of apple, but not of banana. This work asks to what extent word embed- dings can address the challenging task of detect- ing discriminating attributes. On the one hand, word embeddings have proven useful for a wide range of NLP tasks, including semantic similar- ity (Mikolov et al., 2013a; Pennington et al., 2014; Baroni et al., 2014; Upadhyay et al., 2016) and de- tection of lexical semantic relations, either explic- itly by detecting hypernymy, lexical entailment (Baroni et al., 2012; Roller et al., 2014; Turney and Mohammad, 2013), or implicitly using analogies (Mikolov et al., 2013b). On the other hand, de- tecting discriminating attributes requires making fine-grained meaning distinctions, and it is unclear to what extent they can be captured with opaque dense representations. We start our study with unsupervised models. We propose a straightforward approach where pre- dictions are based on a learned threshold for the cosine similarity difference between (w1, d) and …

Content truncated to 500 words