Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3503–3508, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3503 Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space Kawin Ethayarajh∗ Stanford University kawin@stanford.edu Abstract A notable property of word embeddings is that word relationships can exist as linear sub- structures in the embedding space. For exam- ple, gender corresponds to ⃗ woman − ⃗ man and ⃗ queen− ⃗ king. This, in turn, allows word analo- gies to be solved arithmetically: ⃗ king− ⃗ man+ ⃗ woman ≈ ⃗ queen. This property is notable be- cause it suggests that models trained on word embeddings can easily learn such relationships as geometric translations. However, there is no evidence that models exclusively represent relationships in this manner. We document an alternative way in which downstream mod- els might learn these relationships: orthogonal and linear transformations. For example, given a translation vector for gender, we can find an orthogonal matrix R, representing a rota- tion and reflection, such that R( ⃗ king) ≈ ⃗ queen and R( ⃗ man) ≈ ⃗ woman. Analogical reasoning using orthogonal transformations is almost as accurate as using vector arithmetic; using lin- ear transformations is more accurate than both. Our findings suggest that these transforma- tions can be as good a representation of word relationships as translation vectors. 1 Introduction Word embeddings are a cornerstone of current methods in NLP. A notable property of these vec- tors is that word relationships can exist as linear substructures in the embedding space (Mikolov et al., 2013a). For example, gender can be ex- pressed as the translation vectors ⃗ woman − ⃗ man and ⃗ queen − ⃗ king; similarly, past tense can be ex- pressed as ⃗ thought− ⃗ think and ⃗ talked− ⃗ talk. This, in turn, allows word analogies to be solved arith- metically. For example, ‘man is to woman as king is to ?’ can be solved by finding the vector closest to ⃗ king− ⃗ man+ ⃗ woman, which should be ⃗ queen if one excludes the query words. ∗Work partly done at the University of Toronto. Ethayarajh et al. (2019a) proved that when there is no reconstruction error, a word analogy that can be solved arithmetically holds exactly over a set of ordered word pairs iff the co-occurrence shifted PMI is the same for every word pair and across any two word pairs. This means that strict conditions need to be satisfied by the training corpus for a word analogy to hold exactly, and these conditions are not necessarily satisfied by every analogy that makes intuitive sense. For example, most analo- gies involving countries and their currency cannot be solved arithmetically using Wikipedia-trained skipgram vectors (Ethayarajh et al., 2019a). The fact that word relationships can exist as lin- ear substructures is still notable, as it suggests that models trained with embeddings can easily learn …

Content truncated to 500 words