Predicting Word Embeddings Variability

Proceedings of the 7th Joint Conference on Lexical and Computational Semantics (*SEM), pages 154–159 New Orleans, June 5-6, 2018. c⃝2018 Association for Computational Linguistics Predicting Word Embeddings Variability B´en´edicte Pierrejean and Ludovic Tanguy CLLE: CNRS & University of Toulouse Toulouse, France {benedicte.pierrejean,ludovic.tanguy}@univ-tlse2.fr Abstract Neural word embeddings models (such as those built with word2vec) are known to have stability problems: when retraining a model with the exact same hyperparameters, words neighborhoods may change. We propose a method to estimate such variation, based on the overlap of neighbors of a given word in two models trained with identical hyperparam- eters. We show that this inherent variation is not negligible, and that it does not affect every word in the same way. We examine the influ- ence of several features that are intrinsic to a word, corpus or embedding model and provide a methodology that can predict the variability (and as such, reliability) of a word representa- tion in a semantic vector space. 1 Introduction Word embeddings are dense representations of the meaning of words that are efficient and easy to use. Embeddings training methods such as word2vec (Mikolov et al., 2013), are based on neural net- works methods that imply random processes (ini- tialization of the network, sampling, etc.). As such, they display stability problems (Hellrich and Hahn, 2016) meaning that retraining a model with the exact same hyperparameters will give different word representations, with a word possibly having different nearest neighbors from one model to the other. Benchmarks test sets such as WordSim-353 (Finkelstein et al., 2002) are commonly used to evaluate word embeddings since they provide a fast and easy way to quickly evaluate a model (Nayak et al., 2016). However, the instability of word embeddings is not detected by these test sets since only selected pairs of words are evaluated. A model showing instability could get very sim- ilar performance results when evaluated on such benchmarks. Hyperparameters selected when training word embeddings impact the semantic representation of a word. Among these hyperparameters we find some hyperparameters internal to the system such as the architecture used, the size of the context window or the dimensions of the vectors as well as some external hyperparameters such as the corpus used for training (Asr et al., 2016; Baroni et al., 2014; Chiu et al., 2016; Li et al., 2017; Mela- mud et al., 2016; Roberts, 2016). In this work, we adopt a corpus linguistics approach in a simi- lar way to Antoniak and Mimno (2018), Hamilton et al. (2016) and Hellrich and Hahn (2016) mean- ing that observing the semantic representation of a word consists in observing the nearest neighbors of this word. Corpus tools such as Sketch Engine (Kilgarriff et al., 2014) use embeddings trained on several corpora1 to provide users with most similar words as a lexical semantic information on a tar- get word. In order to make accurate observations, it thus seems important to understand the stability of these embeddings. In this paper, we measure the variation that ex- …

Content truncated to 500 words