What a Nerd! Beating Students and Vector Cosine in the ESL and TOEFL Datasets

What a Nerd! Beating Students and Vector Cosine in the ESL and TOEFL Datasets Enrico Santus*, Tin-Shing Chiu*, Qin Lu*, Alessandro Lenci§, Chu-Ren Huang* * The Hong Kong Polytechnic University, Hong Kong esantus@gmail.com, cstschiu@comp.polyu.edu.hk, {qin.lu, churen.huang}@polyu.edu.hk § University of Pisa, Italy alessandro.lenci@unipi.it Abstract In this paper, we claim that Vector Cosine – which is generally considered one of the most efficient unsupervised measures for identifying word similarity in Vector Space Models – can be outperformed by a completely unsupervised measure that evaluates the extent of the intersection among the most associated contexts of two target words, weighting such intersection according to the rank of the shared contexts in the dependency ranked lists. This claim comes from the hypothesis that similar words do not simply occur in similar contexts, but they share a larger portion of their most relevant contexts compared to other related words. To prove it, we describe and evaluate APSyn, a variant of Average Precision that – independently of the adopted parameters – outperforms the Vector Cosine and the co-occurrence on the ESL and TOEFL test sets. In the best setting, APSyn reaches 0.73 accuracy on the ESL dataset and 0.70 accuracy in the TOEFL dataset, beating therefore the non-English US college applicants (whose average, as reported in the literature, is 64.50%) and several state-of-the-art approaches. Keywords: Vector Space Models, VSMs, Distributional Semantic Models, DSMs, Semantic Relations, Words Similarity 1. Introduction Word similarity detection plays an important role in Natural Language Processing (NLP), as it is the backbone of several applications, such as Paraphrasing, Query Expansion, Word Sense Disambiguation, Automatic Thesauri Creation, and so on (Terra and Clarke, 2003). Several approaches have been proposed to measure word similarity (Terra and Clarke, 2003; Jarmasz and Szpakowicz, 2003; Mikolov et al., 2013; Levy et al., 2015; Santus et al. 2016a). Some of them rely on knowledge resources (such as lexicons or semantic networks), while others are corpus-based. The latter approaches generally exploit the Distributional Hypothesis, according to which words that occur in similar contexts also have similar meanings (Harris, 1954). Although these approaches extract statistics from large corpora, they vary in the way they define what has to be considered context (i.e. lexical context, syntactic context, documents, etc.), how the association with such context is measured (e.g. frequency of co-occurrence, association measures like Pointwise Mutual Information, etc.), and how the association with the contexts is used to identify the similarity (Terra and Clarke, 2003; Hearst, 1992; Santus et al., 2014a; Santus et al., 2014b; Santus et al., 2016a). A common way to represent word meaning in NLP is by using vectors to encode the association between the target words and their contexts. The resulting vector space is generally referred as Vector Space Model (VSM) or, more specifically, as Distributional Semantic Model (DSM). In such vector space, word similarity can be calculated by using the Vector Cosine, which measures the angle between the vectors (Turney and Pantel, 2010). Other measures – such as Manhattan Distance, Dice’s Coefficient, Euclidean Distance, Jaccard Similarity …

Content truncated to 500 words