Analyzing the Surprising Variability in Word Embedding Stability Across Languages

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5891–5901 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5891 Analyzing the Surprising Variability in Word Embedding Stability Across Languages Laura Burdick, Jonathan K. Kummerfeld and Rada Mihalcea Computer Science & Engineering University of Michigan, Ann Arbor {lburdick,jkummerf,mihalcea}@umich.edu Abstract Word embeddings are powerful representa- tions that form the foundation of many natu- ral language processing architectures, both in English and in other languages. To gain fur- ther insight into word embeddings, we explore their stability (e.g., overlap between the near- est neighbors of a word in different embedding spaces) in diverse languages. We discuss lin- guistic properties that are related to stability, drawing out insights about correlations with affixing, language gender systems, and other features. This has implications for embedding use, particularly in research that uses them to study language trends. 1 Introduction Word embeddings have become an established part of natural language processing (NLP) (Collobert et al., 2011; Wang et al., 2020a). Stability, defined as the overlap between the nearest neighbors of a word in different embedding spaces, was intro- duced to measure variations in local embedding neighborhoods across changes in data, algorithms, and word properties (Antoniak and Mimno, 2018; Wendlandt et al., 2018). These studies found that many common English embedding spaces are sur- prisingly unstable, which has implications for work that uses embeddings as features in downstream tasks, and work that uses embeddings to study spe- cific properties of language. However, research to date on word embedding stability has been exclusively done on English and so is not representative of all languages. In this work, we explore the stability of word embeddings in a wide range of languages. Better understanding the differences caused by diverse languages will provide a foundation for building embeddings and NLP tools in all languages.1 1 Code is available at https://lit.eecs.umich. edu/downloads.html. In English and other very high resource lan- guages, it has become common practice to use contextualized word embeddings, such as BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019). These algorithms require huge amounts of computa- tional resources and data. For example, it takes 2.5 days to train XLNet with 512 TPU v3 chips. In ad- dition to requiring heavy computational resources, most contextualized embedding algorithms need large amounts of data. BERT uses 3.3 billion words of training data. In contrast to these large corpora, many datasets from low-resource languages are fairly small (Maxwell and Hughes, 2006). To sup- port scenarios where using huge amounts of data and computational resources is not feasible, it is im- portant to continue developing our understanding of context-independent word embeddings, such as word2vec (Mikolov et al., 2013) and GloVe (Pen- nington et al., 2014). These algorithms continue to be used in a wide variety of situations, including the computational humanities (Abdulrahim, 2019; Hellrich et al., 2019) and languages where only small corpora are available (Joshi et al., 2019). In this work, we consider how stability varies for different languages, …

Content truncated to 500 words