Denoising Word Embeddings by Averaging in a Shared Space

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 294–301 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 294 Denoising Word Embeddings by Averaging in a Shared Space Avi Caciularu1 Ido Dagan1 Jacob Goldberger2 1Computer Science Department, Bar-Ilan University 2Faculty of Engineering, Bar-Ilan University avi.c33@gmail.com, dagan@cs.biu.ac.il, jacob.goldberger@biu.ac.il Abstract We introduce a new approach for smoothing and improving the quality of word embed- dings. We consider a method of fusing word embeddings that were trained on the same corpus but with different initializations. We project all the models to a shared vector space using an efficient implementation of the Gen- eralized Procrustes Analysis (GPA) procedure, previously used in multilingual word transla- tion. Our word representation demonstrates consistent improvements over the raw models as well as their simplistic average, on a range of tasks. As the new representations are more stable and reliable, there is a noticeable im- provement in rare word evaluations. 1 Introduction Continuous (non-contextualized) word embeddings have been introduced several years ago as a stan- dard building block for NLP tasks. These models provide efficient ways to learn word representations in a fully self-supervised manner from text corpora, solely based on word co-occurrence statistics. A wide variety of methods now exist for generating word embeddings, with prominent methods includ- ing word2vec (Mikolov et al., 2013a), GloVe (Pen- nington et al., 2014), and FastText (Bojanowski et al., 2017). Recently, contextualized embeddings (Peters et al., 2018; Devlin et al., 2019), replaced the use of non-contextualized embeddings in many settings. Yet, the latter remain the standard choice for typical lexical-semantic tasks, e.g., semantic similarity (Hill et al., 2015), word analogy (Jurgens et al., 2012), relation classification (Barkan et al., 2020a), and paraphrase identification (Meged et al., 2020). These tasks consider the generic meanings of lexical items, given out of context, hence the use of non-contextualized embeddings is appropriate. Notably, FastText was shown to yield state-of-the- art results in most of these tasks (Bojanowski et al., 2017). While word embedding methods proved to be powerful, they suffer from a certain level of noise, introduced by quite a few randomized steps in the embedding generation process, including embed- ding initialization, negative sampling, subsampling and mini-batch ordering. Consequently, different runs would yield different embedding geometries, of varying quality. This random noise might harm most severely the representation of rare words, for which the actual data signal is rather weak (Barkan et al., 2020b). In this paper, we propose denoising word em- bedding models through generating multiple model versions, each created with different random seeds. Then, the resulting representations for each word should be fused effectively, in order to obtain a model with a reduced level of noise. Note, how- ever, that simple averaging of the original word vectors is problematic, since each training session of the algorithm produces embeddings in a differ- ent space. In fact, the objective scores of both word2vec, Glove and FastText are invariant to mul- tiplying all the word embeddings by an orthogonal matrix, hence, the …

Content truncated to 500 words