Factors Influencing the Surprising Instability of Word Embeddings

Proceedings of NAACL-HLT 2018, pages 2092–2102 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Factors Influencing the Surprising Instability of Word Embeddings Laura Wendlandt, Jonathan K. Kummerfeld and Rada Mihalcea Computer Science & Engineering University of Michigan, Ann Arbor {wenlaura,jkummerf,mihalcea}@umich.edu Abstract Despite the recent popularity of word embed- ding methods, there is only a small body of work exploring the limitations of these repre- sentations. In this paper, we consider one as- pect of embedding spaces, namely their sta- bility. We show that even relatively high fre- quency words (100-200 occurrences) are often unstable. We provide empirical evidence for how various factors contribute to the stability of word embeddings, and we analyze the ef- fects of stability on downstream tasks. 1 Introduction Word embeddings are low-dimensional, dense vector representations that capture semantic prop- erties of words. Recently, they have gained tremendous popularity in Natural Language Pro- cessing (NLP) and have been used in tasks as diverse as text similarity (Kenter and De Rijke, 2015), part-of-speech tagging (Tsvetkov et al., 2016), sentiment analysis (Faruqui et al., 2015), and machine translation (Mikolov et al., 2013a). Although word embeddings are widely used across NLP, their stability has not yet been fully evaluated and understood. In this paper, we ex- plore the factors that play a role in the stability of word embeddings, including properties of the data, properties of the algorithm, and properties of the words. We find that word embeddings exhibit substantial instabilities, which can have implica- tions for downstream tasks. Using the overlap between nearest neighbors in an embedding space as a measure of stability (see section 3 below for more information), we ob- serve that many common embedding spaces have large amounts of instability. For example, Figure 1 shows the instability of the embeddings obtained by training word2vec on the Penn Treebank (PTB) (Marcus et al., 1993). As expected, lower fre- quency words have lower stability and higher fre- 20 24 28 212 216 Frequency of Word in PTB (log scale) 100 75 50 25 10 5 2 % Stability (log scale) 1e-04 1e-03 1e-02 1e-01 1 % Words in Particular Frequency Bucket (log scale) Figure 1: Stability of word2vec as a property of fre- quency in the PTB. Stability is measured across ten randomized embedding spaces trained on the training portion of the PTB (determined using language model- ing splits (Mikolov et al., 2010)). Each word is placed in a frequency bucket (x-axis), and each column (fre- quency bucket) is normalized. quency words have higher stability. What is sur- prising however about this graph is the medium- frequency words, which show huge variance in stability. This cannot be explained by frequency, so there must be other factors contributing to their instability. In the following experiments, we explore which factors affect stability, as well as how this stability affects downstream tasks that word embeddings are commonly used for. To our knowledge, this is the first study comprehensively examining the factors behind instability. 2 Related Work …

Content truncated to 500 words