Towards a Gold Standard for Evaluating Danish Word Embeddings

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4754–4763 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4754 Towards a Gold Standard for Evaluating Danish Word Embeddings Nina Skovgaard Schneidermann1, Rasmus Hvingelby2, Bolette Sandford Pedersen1 1Center for Language Technology, University of Copenhagen, Njalsgade 138, 2Alexandra Institute, Njalsgade 76 ninasc91@gmail.com, rasmus.hvingelby@alexandra.dk, bspedersen@hum.ku.dk Abstract This paper presents the process of compiling a model-agnostic similarity gold standard for evaluating Danish word embeddings based on human judgments made by 42 native speakers of Danish. Word embeddings resemble semantic similarity solely by distribution (meaning that word vectors do not reflect relatedness as differing from similarity), and we argue that this generalisation poses a problem in most intrinsic evaluation scenarios. In order to be able to evaluate on both dimensions, our human-generated dataset is therefore designed to reflect the distinction between relatedness and similarity. The goal standard is applied for evaluating the "goodness" of six existing word embedding models for Danish, and it is discussed how a relatively low correlation can be explained by the fact that semantic similarity is substantially more challenging to model than relatedness, and that there seems to be a need for future human judgements to measure similarity in full context and along more than a single spectrum. Keywords: word embeddings, semantic similarity, Danish 1. Introduction ‘You shall know a word by the company it keeps’ is a com- mon phrase used to express the linguistic hypothesis that the meaning of a given word is, at least to some extend, a function of its surrounding context (Lenci, 2008). This assumption, better known as the distributional hypothesis, was earliest expressed by Firth (1957) and has, in spite of its widely debated psycholinguistic validity, been a basis for the linguistic analysis of meaning in certain method- ological paradigms; namely, it has become a cornerstone for methodology in corpus linguistics, as well as the ba- sis for generating computational models of semantic analy- sis using distributed vector representations of words (Gold- berg and Levy, 2014; Mikolov et al., 2013a; Mikolov et al., 2013b). One of the dominant trends in natural language processing (NLP) is the use of neural word embeddings, a non-linear model in which the linguistic input is represented as a vec- tor in a dense vector space. Intuitively, feature vectors, which can be conceived as coordinate points in a vector space, represent words in a vocabulary as points in a dis- tributional space, where each feature encodes a statistical association between the word and its surrounding context as defined by the model. As such, the assumption that se- mantic distances between two words are a function of their distributional similarity is encoded in the vector space as distances between points, typically measured using cosine similarity (Goldberg, 2016). Traditionally, techniques in natural language processing have relied on sparse vector inputs to a linear machine learning model such as logis- tic regression or support vector machines (SVM). In con- trast, word embeddings differ by representing …

Content truncated to 500 words