Improving Distributional Similarity with Lessons Learned from Word Embeddings

Improving Distributional Similarity with Lessons Learned from Word Embeddings Omer Levy Yoav Goldberg Computer Science DepartmentIdo Dagan Bar-Ilan University Ramat-Gan, Israel {omerlevy,yogo,dagan}@cs.biu.ac.il Abstract Recent trends suggest that neural- network-inspired word embedding models outperform traditional count-based distri- butional models on word similarity and analogy detection tasks. We reveal that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter op- timizations, rather than the embedding algorithms themselves. Furthermore, we show that these modifications can be transferred to traditional distributional models, yielding similar gains. In contrast to prior reports, we observe mostly local or insignificant performance differences between the methods, with no global advantage to any single approach over the others. 1 Introduction Understanding the meaning of a word is at the heart of natural language processing (NLP). While a deep, human-like, understanding remains elu- sive, many methods have been successful in recov- ering certain aspects of similarity between words. Recently, neural-network based approaches in which words are “embedded” into a low- dimensional space were proposed by various au- thors (Bengio et al., 2003; Collobert and Weston, 2008). These models represent each word as a d- dimensional vector of real numbers, and vectors that are close to each other are shown to be se- mantically related. In particular, a sequence of pa- pers by Mikolov et al. (2013a; 2013b) culminated in the skip-gram with negative-sampling training method (SGNS): an efficient embedding algorithm that provides state-of-the-art results on various lin- guistic tasks. It was popularized via word2vec, a program for creating word embeddings. A recent study by Baroni et al. (2014) con- ducts a set of systematic experiments compar- ing word2vec embeddings to the more tradi- tional distributional methods, such as pointwise mutual information (PMI) matrices (see Turney and Pantel (2010) and Baroni and Lenci (2010) for comprehensive surveys). These results suggest that the new embedding methods consistently out- perform the traditional methods by a non-trivial margin on many similarity-oriented tasks. How- ever, state-of-the-art embedding methods are all based on the same bag-of-contexts representation of words. Furthermore, analysis by Levy and Goldberg (2014c) shows that word2vec’s SGNS is implicitly factorizing a word-context PMI ma- trix. That is, the mathematical objective and the sources of information available to SGNS are in fact very similar to those employed by the more traditional methods. What, then, is the source of superiority (or per- ceived superiority) of these recent embeddings? While the focus of the presentation in the word- embedding literature is on the mathematical model and the objective being optimized, other factors af- fect the results as well. In particular, embedding algorithms suggest some natural hyperparameters that can be tuned; many of which were already tuned to some extent by the algorithms’ design- ers. Some hyperparameters, such as the number of negative samples to use, are clearly marked as tunable. Other modifications, such as smoothing the negative-sampling distribution, are reported in passing and considered thereafter as part of the al- gorithm. Others still, such as dynamically-sized context windows, are …

Content truncated to 500 words