Evaluating the Stability of Embedding-based Word Similarities

Evaluating the Stability of Embedding-based Word Similarities Maria Antoniak Cornell University maa343@cornell.edu David Mimno Cornell University mimno@cornell.edu Abstract Word embeddings are increasingly being used as a tool to study word associations in specific corpora. However, it is unclear whether such embeddings reflect enduring properties of lan- guage or if they are sensitive to inconsequential variations in the source documents. We find that nearest-neighbor distances are highly sen- sitive to small changes in the training corpus for a variety of algorithms. For all methods, including specific documents in the training set can result in substantial variations. We show that these effects are more prominent for smaller training corpora. We recommend that users never rely on single embedding models for distance calculations, but rather average over multiple bootstrap samples, especially for small corpora. 1 Introduction Word embeddings are a popular technique in natural language processing (NLP) in which the words in a vocabulary are mapped to low-dimensional vectors. Embedding models are easily trained—several imple- mentations are publicly available—and relationships between the embedding vectors, often measured via cosine similarity, can be used to reveal latent seman- tic relationships between pairs of words. Word em- beddings are increasingly being used by researchers in unexpected ways and have become popular in fields such as digital humanities and computational social science (Hamilton et al., 2016; Heuser, 2016; Phillips et al., 2017). Embedding-based analyses of semantic similarity can be a robust and valuable tool, but we find that standard methods dramatically under-represent the variability of these measurements. Embedding algo- rithms are much more sensitive than they appear to factors such as the presence of specific documents, the size of the documents, the size of the corpus, and even seeds for random number generators. If users do not account for this variability, their conclusions are likely to be invalid. Fortunately, we also find that simply averaging over multiple bootstrap samples is sufficient to produce stable, reliable results in all cases tested. NLP research in word embeddings has so far fo- cused on a downstream-centered use case, where the end goal is not the embeddings themselves but performance on a more complicated task. For exam- ple, word embeddings are often used as the bottom layer in neural network architectures for NLP (Ben- gio et al., 2003; Goldberg, 2017). The embeddings’ training corpus, which is selected to be as large as possible, is only of interest insofar as it generalizes to the downstream training corpus. In contrast, other researchers take a corpus- centered approach and use relationships between em- beddings as direct evidence about the language and culture of the authors of a training corpus (Bolukbasi et al., 2016; Hamilton et al., 2016; Heuser, 2016). Embeddings are used as if they were simulations of a survey asking subjects to free-associate words from query terms. Unlike the downstream-centered approach, the corpus-centered approach is based on direct human analysis of nearest neighbors to embed- ding vectors, and the training corpus is not simply an off-the-shelf convenience but rather the central object of …

Content truncated to 500 words