Word Embeddings as Metric Recovery in Semantic Spaces

Word Embeddings as Metric Recovery in Semantic Spaces Tatsunori B. Hashimoto, David Alvarez-Melis and Tommi S. Jaakkola CSAIL, Massachusetts Institute of Technology {thashim, davidam, tommi}@csail.mit.edu Abstract Continuous word representations have been remarkably useful across NLP tasks but re- main poorly understood. We ground word embeddings in semantic spaces studied in the cognitive-psychometric literature, taking these spaces as the primary objects to recover. To this end, we relate log co-occurrences of words in large corpora to semantic similarity assessments and show that co-occurrences are indeed consistent with an Euclidean semantic space hypothesis. Framing word embedding as metric recovery of a semantic space uni- fies existing word embedding algorithms, ties them to manifold learning, and demonstrates that existing algorithms are consistent metric recovery methods given co-occurrence counts from random walks. Furthermore, we propose a simple, principled, direct metric recovery al- gorithm that performs on par with the state-of- the-art word embedding and manifold learning methods. Finally, we complement recent fo- cus on analogies by constructing two new in- ductive reasoning datasets—series completion and classification—and demonstrate that word embeddings can be used to solve them as well. 1 Introduction Continuous space models of words, objects, and sig- nals have become ubiquitous tools for learning rich representations of data, from natural language pro- cessing to computer vision. Specifically, there has been particular interest in word embeddings, largely due to their intriguing semantic properties (Mikolov et al., 2013b) and their success as features for down- stream natural language processing tasks, such as named entity recognition (Turian et al., 2010) and parsing (Socher et al., 2013). The empirical success of word embeddings has prompted a parallel body of work that seeks to better understand their properties, associated estimation al- gorithms, and explore possible revisions. Recently, Levy and Goldberg (2014a) showed that linear lin- guistic regularities first observed with word2vec extend to other embedding methods. In particu- lar, explicit representations of words in terms of co- occurrence counts can be used to solve analogies in the same way. In terms of algorithms, Levy and Goldberg (2014b) demonstrated that the global min- imum of the skip-gram method with negative sam- pling of Mikolov et al. (2013b) implicitly factorizes a shifted version of the pointwise mutual informa- tion (PMI) matrix of word-context pairs. Arora et al. (2015) explored links between random walks and word embeddings, relating them to contextual (prob- ability ratio) analogies, under specific (isotropic) as- sumptions about word vectors. In this work, we take semantic spaces stud- ied in the cognitive-psychometric literature as the prototypical objects that word embedding algo- rithms estimate. Semantic spaces are vector spaces over concepts where Euclidean distances between points are assumed to indicate semantic similar- ities. We link such semantic spaces to word co-occurrences through semantic similarity assess- ments, and demonstrate that the observed co- occurrence counts indeed possess statistical proper- ties that are consistent with an underlying Euclidean space where distances are linked to semantic simi- larity. 273 Transactions of the Association for Computational Linguistics, vol. 4, pp. 273–286, 2016. Action …

Content truncated to 500 words