Paraphrases do not explain word analogies

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2129–2134 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2129 Paraphrases do not explain word analogies Louis Fournier Cognitive Machine Learning CNRS-EHESS-INRIA PSL Research University Paris, France louis.fournier@polytechnique.edu Ewan Dunbar Cognitive Machine Learning CNRS-EHESS-INRIA PSL Research University Paris, France and University of Toronto Toronto, Canada ewan.dunbar@utoronto.ca Abstract Many types of distributional word embeddings (weakly) encode linguistic regularities as di- rections (the difference between jump and jumped will be in a similar direction to that of walk and walked, and so on). Several at- tempts have been made to explain this fact. We respond to Allen and Hospedales’ recent (ICML, 2019) theoretical explanation, which claims that word2vec and GloVe will encode linguistic regularities whenever a specific re- lation of paraphrase holds between the four words involved in the regularity. We demon- strate that the explanation does not go through: the paraphrase relations needed under this ex- planation do not hold empirically. 1 Introduction The study of linguistic regularities in distributional word embeddings—that the difference vector calcu- lated between the vectors jump and jumped shows a similar direction to that of walk and walked, and so on—has been both stimulating and controversial. While a number of such regularities appear to hold, across a number of different kinds of embeddings, the standard 3COSADD analogy test used to mea- sure the presence of these regularities has come under fire for confounding analogical regularities with unrelated properties of semantic embeddings. It is thus important to note that several papers have proposed theoretical explanations for why linguis- tic regularities should hold in distributional word embeddings. Particularly in light of the controver- sies over linguistic regularities, it is important to examine the soundness of these arguments. Allen and Hospedales (2019) develop such an explanation by linking the semantic definition of an analogy to paraphrases. In the sense of Git- tens et al. (2017), paraphrases are sets of words which are semantically and distributionally closely equivalent to another word or set of words—for ex- ample, king may be paraphrased by {man, royal}. Allen and Hospedales argue that the standard anal- ogy criterion, that king - man + woman = queen, is equivalent to a criterion whereby {king, woman} paraphrases {man, queen}. With this in mind, it becomes possible to rewrite the arithmetic analogy criterion in terms of vectors encoding the pointwise mutual information (PMI) between words and their contexts, and to decompose the error in the anal- ogy equality into several components, including a paraphrase error term measuring the degree to which the critical paraphrase holds. Making use of an assumption that the word2vec embedding is a linear transformation of the PMI matrix, they argue that results in terms of PMI apply to word vectors. Thus, under their explanation, a major part of suc- cess on an analogy a − a∗ + b∗ = b is due to a, b∗ and a∗, b being close distributional paraphrases. We first review …

Content truncated to 500 words