Word Embeddings, Analogies, and Machine Learning: Beyond king - man + woman = queen

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3519–3530, Osaka, Japan, December 11-17 2016. Word Embeddings, Analogies, and Machine Learning: Beyond King - Man + Woman = Queen Aleksandr Drozd†, Anna Gladkova‡, Satoshi Matsuoka† † Tokyo Institute of Technology, Meguro-ku, Tokyo 152-8550, Japan alex@smg.is.titech.ac.jp, matsu@is.titech.ac.jp ‡ The University of Tokyo, Meguro-ku, Tokyo 153-8902 Japan gladkova@phiz.c.u-tokyo.ac.jp Abstract Solving word analogies became one of the most popular benchmarks for word embeddings on the assumption that linear relations between word pairs (such as king:man :: woman:queen) are indicative of the quality of the embedding. We question this assumption by showing that the information not detected by linear offset may still be recoverable by a more sophisticated search method, and thus is actually encoded in the embedding. The general problem with linear offset is its sensitivity to the idiosyncrasies of individual words. We show that simple averaging over multiple word pairs improves over the state-of-the-art. A further improvement in accuracy (up to 30% for some embeddings and relations) is achieved by combining cosine similarity with an estimation of the extent to which a candidate answer belongs to the correct word class. In addition to this practical contribution, this work highlights the problem of the interaction between word embeddings and analogy retrieval algorithms, and its implications for the evaluation of word embeddings and the use of analogies in extrinsic tasks. 1 Introduction Discovering analogical relations is currently one of the most popular benchmarks for word embeddings. This trend started after (Mikolov et al., 2013b) showed that proportional analogies (a is to b as c is to d) can be solved by finding the vector closest to the hypothetical vector calculated as c - a + b (e.g. king - man + woman = queen). Many subsequent studies used this approach to evaluate the performance of word embeddings with the Google test set (Mikolov et al., 2013a); the top current result is over 80% accuracy (Pennington et al., 2014). The assumption is that a “good” word embedding encodes linguistic relations in such a way that they are identifiable via linear vector offset (see section 2). Analogies are interesting not only as a benchmark, but also potentially as a method for discovering linguistic relations (Turney, 2008). They are already used for morphological analysis (Lavall´ee and Langlais, 2010), word sense disambiguation (Federici et al., 1997), semantic search (Cohen et al., 2015), and even for broad-range detection of both morphological and semantic features (Lepage and Goh, 2009). However, Mikolov’s study was a demonstration of how word embeddings capture linguistic relations, rather than a proposal of linear vector offset as a method for their discovery. It was later shown to not work as well for a wider range of relations (K¨oper et al., 2015; Gladkova et al., 2016). This study questions the underlying assumption that linguistic relations should translate to linear re- lations between vectors rather than a more complex correspondence pattern. We show that relations not detected by vector offset may be recoverable …

Content truncated to 500 words