The (too Many) Problems of Analogical Reasoning with Word Vectors

Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 135–148, Vancouver, Canada, August 3-4, 2017. c⃝2017 Association for Computational Linguistics The (Too Many) Problems of Analogical Reasoning with Word Vectors Anna Rogers Dept. of Computer Science University of Massachusetts Lowell Lowell, MA, USA arogers@cs.uml.edu Aleksandr Drozd Global Scient. Inf. and Comput. Center Tokyo Institute of Technology Tokyo, Japan alex@smg.is.titech.ac.jp Bofang Li School of Information Renmin University of China Beijing, China libofang@ruc.edu.cn Abstract This paper explores the possibilities of analogical reasoning with vector space models. Given two pairs of words with the same relation (e.g. man:woman :: king:queen), it was proposed that the off- set between one pair of the corresponding word vectors can be used to identify the unknown member of the other pair (−−→ king − −−→ man + −−−−−→ woman = ?−−−→ queen). We ar- gue against such “linguistic regularities” as a model for linguistic relations in vector space models and as a benchmark, and we show that the vector offset (as well as two other, better-performing methods) suffers from dependence on vector similarity. 1 Introduction This paper considers the phenomenon of “vector- oriented reasoning” via linear vector offset in vector space models (VSMs) (Mikolov et al., 2013c,a). Given two pairs of words with the same linguistic relation (woman:man :: king:queen), it has been proposed that the offset between one pair of word vectors can be used to identify the un- known member of a different pair of words via solving proportional analogy problems (−−→ king − −−→ man + −−−−−→ woman = ?−−−→ queen), as shown in Fig. 1. We will refer to this method as 3CosAdd. This approach attracted a lot of attention, both as the “poster child” of word embeddings, and for its potential practical utility. Given the vi- tal role that analogical reasoning plays in human cognition for discovering new knowledge and un- derstanding new concepts, automated analogical reasoning could become a game-changer in many fields, providing a universal mechanism for detect- ing linguistic relations (Turney, 2008) and word sense disambiguation (Federici et al., 1997). It is already used in many downstream NLP tasks, such as splitting compounds (Daiber et al., 2015), se- mantic search (Cohen et al., 2015), cross-language relational search (Duc et al., 2012), to name a few. Figure 2: Left panel shows vector offsets for three word pairs illustrating the gender relation. Right panel shows a different projection, and the singular/plural relation for two words. In high-dimensional space, multiple relations can be embedded for a single word. provided. We have explored several related meth- ods and found that the proposed method performs well for both syntactic and semantic relations. We note that this measure is qualitatively similar to rela- tional similarity model of (Turney, 2012), which pre- dicts similarity between members of the word pairs (xb, xd), (xc, xd) and dis-similarity for (xa, xd). 6 Experimental Results To evaluate the vector offset method, we used vectors generated by the RNN toolkit of Mikolov (2012). Vectors of …

Content truncated to 500 words