Word Re-Embedding via Manifold Dimensionality Retention

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 321–326 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Word Re-Embedding via Manifold Dimensionality Retention Souleiman Hasan and Edward Curry Lero- The Irish Software Research Centre National University of Ireland, Galway {souleiman.hasan, edward.curry}@lero.ie Abstract Word embeddings seek to recover a Eu- clidean metric space by mapping words into vectors, starting from words co- occurrences in a corpus. Word embed- dings may underestimate the similarity be- tween nearby words, and overestimate it between distant words in the Euclidean metric space. In this paper, we re-embed pre-trained word embeddings with a stage of manifold learning which retains dimen- sionality. We show that this approach is theoretically founded in the metric recov- ery paradigm, and empirically show that it can improve on state-of-the-art embed- dings in word similarity tasks 0.5 − 5.0% points depending on the original space. 1 Introduction Concepts have been hypothesized in the cognitive psychometric literature as points in a Euclidean metric space, with empirical support from human judgement experiments (Rumelhart and Abraham- son, 1973; Sternberg and Gardner, 1983). Word embeddings, such as GloVe (Pennington et al., 2014a) and Word2Vec (Mikolov et al., 2013), harvest observed features of the latent Euclidean space such as words co-occurrence counts in a corpus and turn words into dense vectors of a few hundred dimensions. Word embeddings have proved useful in downstream NLP tasks such as Part of Speech Tagging (Collobert, 2011), Named Entity Recognition (Turian et al., 2010), and Ma- chine Translation (Devlin et al., 2014). However, the potential of word embeddings and further im- provements remain a research question. When comparing word pairs similarities ob- tained from word embeddings, to word pairs sim- ilarities obtained from human judgement, it is ob- served that word embeddings slightly underesti- mate the similarity between similar words, and overestimate the similarity between distant words. For example, in the WS353 (Finkelstein et al., 2001) word similarity ground truth: sim(“shore”, “woodland”) = 3.08 < sim(“physics”, “proton”) = 8.12 However, the use of GloVe 42B 300d embed- ding with cosine similarity (see Section 4) yields the opposite order: sim(“shore”, “woodland”) = 0.36 > sim(“physics”, “proton”) = 0.33 Re-embedding the space using a manifold learning stage can rectify this. Manifold learning works by estimating the distance between nearby words using direct similarity assignment in a lo- cal neighbourhood, while distance between far- away words is approximated by multiple neigh- bourhoods based on the manifold shape. This ob- servation forms the basis for the rest of this paper. For instance, using Locally Linear Embedding (LLE) (Roweis and Saul, 2000) on top of GloVe, as described in this paper, can recover the right pairs order yielding: sim(“shore”, “woodland”) = 0.08 < sim(“physics”, “proton”) = 0.25 Hashimoto et al. (Hashimoto et al., 2016) put word embeddings under a paradigm which seeks to recover the underlying Euclidean metric seman- tic space. In this paradigm, word embeddings land into a space where a Euclidean metric can be used. They show …

Content truncated to 500 words