Taxonomy Beats Corpus in Similarity Identification, but Does It Matter?

Proceedings of Recent Advances in Natural Language Processing, pages 346–354, Hissar, Bulgaria, Sep 7–9 2015. Taxonomy Beats Corpus in Similarity Identification, but Does It Matter? Minh Ngoc Le VU University Amsterdam m.n.le@vu.nl Antske Fokkens VU University Amsterdam antske.fokkens@vu.nl Abstract We present extensive evaluations compar- ing the performance of taxonomy-based and corpus-based approaches on SimLex- 999. The results confirm our hypothesis that taxonomy-based approaches are more suitable to identify similarity. We intro- duce two new measures of evaluation that show that all measures perform well on a coarse-grained evaluation and that it is not always clear which approach is most suit- able when a similarity score is used as a threshold. This leads us to conclude that the inferior performance of corpus-based approaches may not (always) matter. 1 Introduction Similarity measures are used in a wide variety of Natural Language Processing (NLP) tasks (see Pilehvar et al. (2013), among others for examples). They may be used, e.g. to increase coverage of an approach by using information from similar words for unseen data, or to establish average similarity between a question and a potential answer. Due to its importance, similarity measures have received steady attention in computational linguis- tics. There are two widely followed, but different, schools: taxonomy-based approaches and distri- butional, or corpus-based, approaches. Apart from a few exceptions, these approaches have mostly been studied separately. Our main goal is to examine how the ap- proaches perform when identifying true similar- ity, in contrast to the more general relatedness, which also includes association, between word- pairs. We evaluate the approaches on the new gold-standard SimLex-999 (Hill et al., 2014b). We compare taxonomy-based approaches that use WordNet (Fellbaum, 1998) to the corpus-based approaches that performed best on SimLex-999 in Hill et al. (2014a). We hypothesize that taxonomy- based approaches outperform corpus-based ap- proaches on a true similarity set, because corpus- based approaches tend to mix-up similarity and as- sociation. We carry out several evaluations which investi- gate (i) the difference in performance on pure sim- ilarity sets and sets that combine similarity and association, (ii) the influence of associative pairs while identifying true similarity, and (iii) various evaluation metrics that compare similarity mea- sures to the gold standard of SimLex-999. We perform more than one evaluation metric for two reasons. First, different ranking coeffi- cients can lead to a completely different outcome when evaluating similarity scores (Fokkens et al., 2013). Second, we want to gain more insight into the differences between individual measures. To do so, we introduced two new, more flexible, eval- uation methods which reveal high results for all similarity measures. We argue that these new eval- uations provide a better insight into how suitable similarity measures are to be used in NLP tasks than the commonly used Spearman’s correlation (henceforth Spearman ρ). Our results show that most of the evaluations confirm our hypothesis. The few cases where corpus-based methods outperformed taxonomy- based approaches reveal much smaller differences than the many cases where taxonomy-based ap- proaches have higher results. However, …

Content truncated to 500 words