Analogies minus analogy test: measuring regularities in word embeddings

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 365–375 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 365 Analogies minus analogy test: measuring regularities in word embeddings Louis Fournier 1 louis.fournier@polytechnique.edu Emmanuel Dupoux 1 2 1Cognitive Machine Learning (ENS-CNRS-EHESS-INRIA-PSL Research University), France 2Facebook A.I. Research, Paris, France 3Laboratoire de Linguistique Formelle (CNRS-Paris Diderot-Sorbonne Paris Cite), France 4University of Toronto, Toronto, Canada emmanuel.dupoux@gmail.com Ewan Dunbar 1 3 4 ewan.dunbar@utoronto.ca Abstract Vector space models of words have long been claimed to capture linguistic regulari- ties as simple vector translations, but prob- lems have been raised with this claim. We decompose and empirically analyze the clas- sic arithmetic word analogy test, to motivate two new metrics that address the issues with the standard test, and which distinguish be- tween class-wise offset concentration (simi- lar directions between pairs of words drawn from different broad classes, such as France– London, China–Ottawa, ...) and pairing con- sistency (the existence of a regular transforma- tion between correctly-matched pairs such as France:Paris::China:Beijing). We show that, while the standard analogy test is flawed, sev- eral popular word embeddings do nevertheless encode linguistic regularities. 1 Introduction Vector semantic models saw a surge in interest after embeddings trained under the word2vec architec- ture (Mikolov et al., 2013a) were shown to encode linguistic regularities (Mikolov et al., 2013b). The demonstration relied on the arithmetic analogy test: relations such as king + woman − man ≈ queen were shown to hold for a variety of semantic and grammatical relations. Evaluation of word embed- dings on analogy tests and training on related loss functions remains current. There is also continued interest in theoretically grounding the success of distributional embeddings on these tests (Allen and Hospedales, 2019; Ethayarajh et al., 2019). There is, however, a substantial literature point- ing to problems with word analogies (see Section 2), leading to the conclusion that word analogies are fragile in practice, and sometimes going so far as to imply that the positive results were erroneous. These critiques have been ambiguous as to whether the problem is that the embeddings in question do not really encode the relevant linguistic structure, or whether the issue is merely that the arithmetic analogy test as it is standardly defined, is flawed. The current paper confirms that there are serious problems with the standard analogy test, as it con- founds three different properties of word vectors while purporting to measure only one: class-wise offset concentration (similar directions between pairs of words drawn from different broad classes, such as France–London, China–Ottawa), within- pair similarity between test words (for example, the similarity between Paris and France), as well as the pairing consistency the test sets out to mea- sure (the presence of a regular direction encoding relations such as capital-of: France–Paris, China– Beijing). We give an algebraic decomposition of the standard analogy test that explains previous neg- ative results in terms of within-pair similarity. Us- ing new measures, we show that, in practice, offset concentration, rather than true pairing consistency, …

Content truncated to 500 words