On the Correlation of Word Embedding Evaluation Metrics

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4789–4797 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4789 On the Correlation of Word Embedding Evaluation Metrics François Torregrossa, Vincent Claveau, Nihel Kooli, Guillaume Gravier, Robin Allesiardo Solocal-IRISA, IRISA-CNRS, Solocal, IRISA-CNRS, Solocal Rennes ftorregrossa@solocal.com, vincent.claveau@irisa.fr, nkooli@solocal.com, guillaume.gravier@irisa.fr, rallesiardo@solocal.com Abstract Word embeddings intervene in a wide range of natural language processing tasks. These geometrical representations are easy to manipulate for automatic systems. Therefore, they quickly invaded all areas of language processing. While they surpass all predecessors, it is still not straightforward why and how they do so. In this article, we propose to investigate all kind of evaluation metrics on various datasets in order to discover how they correlate with each other. Those correlations lead to 1) a fast solution to select the best word embeddings among many others, 2) a new criterion that may improve the current state of static Euclidean word embeddings, and 3) a way to create a set of complementary datasets, i.e. each dataset quantifies a different aspect of word embeddings. 1. Introduction Word embeddings are continuous vector representations of word paradigmatics and syntagmatics. Since they capture multiple high-level characteristics of language, their evalua- tion is particularly difficult: it usually consists of quantifying their performance on various tasks. This process is thorny because the outcome value does not explain entirely the complexity of these models. In other words, a model per- forming well under a specific evaluation might poorly work for a different one (Schnabel et al., 2015). As an exam- ple, some word embedding evaluations promote comparison of embeddings with human judgement while others favour embeddings behaviour analysis on downstream tasks, as pointed by Schnabel et al. (2015). In this work, we propose to investigate correlations between numerous evaluations for word embedding. We restrict the study to FastText embeddings introduced by Bojanowski et al. (2017), but this methodology can be applied to other kinds of word embedding techniques. The understanding of evaluation correlations may provide several useful tools: • Strongly correlated evaluations raise a question on the relevance of performing these. Actually, it could be possible to only keep one evaluation among the cor- related evaluation set, since its score would directly affect the score of others. Therefore, it could reduce the number of needed evaluations. • Inexpensive evaluation processes correlated with time- consuming ones could be helpful to speed up optimisa- tion of hyper-parameters. Indeed, they could be used to bypass those demanding steps, thus, saving time. • Some evaluations do not require any external data since they look into global structure of vectors as presented in Tifrea et al. (2018) and Houle et al. (2012). If related to other tasks, these data-free metrics could be incorporated into the optimisation process in order to improve the performance on related tasks. The article is organised as follows. Section 2 compares our proposed methodology to the current state of the art of word embeddings evaluation. Section 3 introduces …

Content truncated to 500 words