UNBNLP at SemEval-2018 Task 10: Evaluating unsupervised approaches to capturing discriminative attributes

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 1013–1016 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics UNBNLP at SemEval-2018 Task 10: Evaluating unsupervised approaches to capturing discriminative attributes Milton King and Ali Hakimi Parizi and Paul Cook Faculty of Computer Science, University of New Brunswick Fredericton, NB E3B 5A3, Canada milton.king@unb.ca, ahakimi@unb.ca, paul.cook@unb.ca Abstract In this paper we present three unsupervised models for capturing discriminative attributes based on information from word embed- dings, WordNet, and sentence-level word co- occurrence frequency. We show that, of these approaches, the simple approach based on word co-occurrence performs best. We fur- ther consider supervised and unsupervised ap- proaches to combining information from these models, but these approaches do not improve on the word co-occurrence model. 1 Introduction In the task of capturing discriminative attributes, a system is presented with three words, and must determine whether the third word — the attribute — characterizes the first word, but not the second. For example, for the triple (chicken,bread,legs), legs is a discriminative at- tribute because chickens typically have legs, but bread typically does not. On the other hand, for the triple (mother,woman,female), female is not a discriminative attribute because both mothers and women are typically female. In the case of the triple (brush,chocolate,chicken), chicken is not a discriminative attribute because there is no clear relationship between chicken and brushes, or be- tween chicken and chocolate. In this paper we focus primarily on unsuper- vised approaches to the task of capturing discrim- inative attributes. We consider three unsupervised models drawing on information from word embed- dings, WordNet (Fellbaum, 1998), and sentence- level word co-occurrence frequency. We then con- sider three approaches to combining information from these models: one unsupervised majority vote approach, and two supervised approaches. Somewhat surprisingly, we achieve our best F1 score of 0.61 with the remarkably simple approach based on word co-occurrence. None of the ap- proaches to model combination improve over this. 2 Base models In this section, we discuss three unsupervised models for identifying discriminative attributes that incorporate information from word embed- dings, WordNet, and word co-occurrences. We refer to these models as “base models”. In Section 3 we describe unsupervised and super- vised approaches to combining these base models. Throughout the description of our models we refer to the words in the triples in the dataset as word1, word2, and attribute, respectively. 2.1 Word2vec If an attribute is a discriminative attribute for word1, then we hypothesize that word1 and the attribute will be more semantically similar than word2 and the attribute. We use similarity of word embeddings as a proxy for semantic similarity. We train word2vec’s skip-gram model (Mikolov et al., 2013) on a snapshot of English Wikipedia from 1 September 2015 containing roughly 2.6 billion tokens, tokenized using the tokenizer avail- able in the Stanford CoreNLP tools (Manning et al., 2014).1 We use a window size of ± 8 and 300 dimensions. We remove all words that occur less than 15 times in …

Content truncated to 500 words