BomJi at SemEval-2018 Task 10: Combining Vector-, Pattern- and Graph-based Information to Identify Discriminative Attributes

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 990–994 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics BomJi at SemEval-2018 Task 10: Combining Vector-, Pattern- and Graph-based Information to Identify Discriminative Attributes Enrico Santus1, Chris Biemann2, Emmanuele Chersoni3 esantus@mit.edu, biemann@informatik.uni-hamburg.de, emmanuelechersoni@gmail.com 1 Massachussetts Institute of Technology, 2 Universit¨at Hamburg, 3 Aix-Marseille University Abstract This paper describes BomJi, a supervised sys- tem for capturing discriminative attributes in word pairs (e.g. yellow as discriminative for banana over watermelon). The system relies on an XGB classifier trained on carefully engi- neered graph-, pattern- and word embedding- based features. It participated in the SemEval- 2018 Task 10 on Capturing Discriminative At- tributes, achieving an F1 score of 0.73 and ranking 2nd out of 26 participant systems. 1 Introduction The recent introduction of popular software packages for training neural word embeddings (Mikolov et al., 2013a,b; Levy and Goldberg, 2014) has led to an increase of the number of stud- ies dedicated to lexical similarity and to remark- able performance improvements on related tasks (Baroni et al., 2014). However, the validity of similarity estimation as the only benchmark for semantic representa- tions has been questioned, for several reasons. One for all, most evaluation datasets provide human-elicited similarity scores, with the conse- quences that the ratings are subjective and the performance of some automated systems is al- ready above the upper bound of the inter-annotator agreement (Batchkarov et al., 2016; Faruqui et al., 2016; Santus et al., 2016a). Originally proposed as an alternative bench- mark for Distributional Semantic Models (DSMs), the Discriminative Attributes task focuses in- stead on the extraction of semantic differences between lexical meanings (Krebs and Paperno, 2016): given two words and an attribute (i.e., a discrete semantic feature), a system has to pre- dict whether the attribute describes a difference between the corresponding concepts or not (e.g. wing is an attribute of plane, but not of helicopter). Since even related words may differ for some non-shared attributes (e.g. hypernyms and hy- ponyms), the ability of automatically recognize discriminative features would be an extremely useful addition for the creation of ontologies and other types of lexical resources and would make machine decisions interpretable, enabling human validation (Biemann et al., 2018). Moreover, one can think to applications to many other NLP do- mains, such as machine translation and dialogue systems (Krebs and Paperno, 2016). In the present contribution, we describe the BomJi classification system, which we used for the identification of discriminative features be- tween concept pairs. According to the official evaluation results provided by the organizers1, our system ranked second out of 26 participants. Our score, F1 = 0.73 lags slightly behind the best score of 0.75. After the evaluation period, we run further experiments including all investigated fea- tures and found that the system can achieve up to 0.75 F1 score. 2 Capturing Discriminative Attributes 2.1 Task and Dataset Description The task of capturing discriminative attributes be- tween words can be described as a binary classi- fication task, …

Content truncated to 500 words