Deconstructing word embedding algorithms

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8479–8484, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8479 Deconstructing word embedding algorithms Kian Kenyon-Dean†∗ BMO AI Capabilities Team Bank of Montreal - Toronto, Ontario kian.kenyon-dean@bmo.com Edward Newell∗, Jackie Chi Kit Cheung Mila - Qu´ebec AI Institute McGill University - Montr´eal, Qu´ebec edward.newell@gmail.com jcheung@cs.mcgill.ca Abstract Word embeddings are reliable feature repre- sentations of words used to obtain high qual- ity results for various NLP applications. Un- contextualized word embeddings are used in many NLP tasks today, especially in resource- limited settings where high memory capacity and GPUs are not available. Given the histor- ical success of word embeddings in NLP, we propose a retrospective on some of the most well-known word embedding algorithms. In this work, we deconstruct Word2vec, GloVe, and others, into a common form, unveiling some of the common conditions that seem to be required for making performant word em- beddings. We believe that the theoretical find- ings in this paper can provide a basis for more informed development of future models. 1 Introduction The advent of efficient uncontextualized word em- bedding algorithms (e.g., Word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014)) marked a historical breakthrough in NLP. Countless re- searchers employed word embeddings in new mod- els to improve results on a multitude of NLP prob- lems. In this work, we provide a retrospective anal- ysis of these groundbreaking models of the past, which simultaneously offers theoretical insights for how future models can be developed and under- stood. We build on the theoretical work of Levy and Goldberg (2014), proving that their findings on the relationship between pointwise mutual in- formation (PMI) and word embeddings go beyond Word2vec and singular value decomposition. In particular, we generalize several word embed- ding algorithms into a common form by proposing the low rank embedder framework. We decon- struct each algorithm into its constituent parts, and ∗ Kian and Edward contributed equally. † This work was pursued while Kian was a member of Mila. find that, despite their many different hyperparam- eters, the algorithms collectively intersect upon the following two key design features. First, vector- covector dot products are learned to approximate PMI statistics in the corpus. Second, modulation of the loss gradient, directly or indirectly, is necessary to balance weak and strong signals arising from the highly imbalanced distribution of corpus statistics. These findings can provide an informed basis for future development of both new embedding algorithms and deep contextualized models. 2 Fundamental concepts We begin by formally defining embeddings, their vectors and covectors (also known as “input” and “output” vectors (Rong, 2014; Nalisnick et al., 2016)), and pointwise mutual information (PMI). Embedding. In general topology, an embedding is understood as an injective structure preserving map, f : X → Y , between two mathematical struc- tures X and Y . A word embedding algorithm (f) learns an inner-product space (Y ) to preserve a lin- guistic structure within a reference corpus of …

Content truncated to 500 words