Evaluation of Greek Word Embeddings

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2543–2551 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2543 Evaluation of Greek Word Embeddings Stamatis Outsios1, Christos Karatsalos1, Konstantinos Skianis2, Michalis Vazirgiannis1,2 1Athens University of Economics and Business, 2’Ecole Polytechnique Greece, France {soutsios, ckarats, skianis.konstantinos, michalis.vazirgiannis}@gmail.com Abstract Since word embeddings have been the most popular input for many NLP tasks, evaluating their quality is critical. Most research efforts are focusing on English word embeddings. This paper addresses the problem of training and evaluating such models for the Greek language. We present a new word analogy test set considering the original English Word2vec analogy test set and some specific linguistic aspects of the Greek language as well. Moreover, we create a Greek version of WordSim353 test collection for a basic evaluation of word similarities. Produced resources are available for download. We test seven word vector models and our evaluation shows that we are able to create meaningful representations. Last, we discover that the morphological complexity of the Greek language and polysemy can influence the quality of the resulting word embeddings. Keywords: word embeddings, evaluation, less-resourced languages 1. Introduction Many unsupervised learning techniques have been pro- posed to obtain representations of words from text. Word embeddings, have been found to provide meaningful repre- sentations for words in an efficient way, so that they have become very common in many NLP tasks. In most cases word embeddings are obtained as a product of training neural network-based models. Language mod- elling is a typical NLP task, where the objective is to predict the probability of the distribution over the “next” word. A word embedding is a vector that has a finite dimension, with the value of each dimension being a feature that weights the relation of the word with a “latent” aspect of the lan- guage. These features are jointly learned during unsuper- vised learning, with plain text data as input, which is not annotated. This principle is known as the distributional hy- pothesis (Harris, 1954). The direct implication of this hy- pothesis is that the word meaning is related to the context where it usually occurs, so it is possible to compare the meanings of two words by applying statistical comparisons of their contexts. All these implications were confirmed by empirical tests carried out on human groups in (Rubenstein and Goodenough, 1965; Charles, 2000). Word embeddings are produced by performing unsuper- vised learning, so their evaluation is critical. Much has been investigated about word embeddings of English words and phrases, but only little attention has been dedicated to other languages. The main objective of this work is to explore the behavior of state-of-the-art word embedding methods on Greek, which is a language that is characterized by a very rich morphology. We introduce a new test set for the word analogy task that inspects syntactic, morphosyntactic and semantic proper- ties of Greek words and phrases. The proposed evaluation scheme is based on word analogies that …

Content truncated to 500 words