Multilingual Culture-Independent Word Analogy Datasets

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4074–4080 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4074 Multilingual Culture-Independent Word Analogy Datasets Matej Ulˇcar1, Kristiina Vaik2,4, Jessica Lindstr¨om3, Milda Dailid˙enait˙e4,5, Marko Robnik-ˇSikonja1 1University of Ljubljana, Faculty of Computer and Information Science, 2Texta O ¨U, Tallinn, Estonia, 3University of Helsinki, Department of Finnish, Finno-Ugrian and Scandinavian Studies, 4University of Tartu, Faculty of Arts and Humanities, 5University of Latvia, Livonian institute, 1Veˇcna pot 113, SI-1000 Ljubljana, Slovenia {matej.ulcar, marko.robnik}@fri.uni-lj.si, kristiina.vaik@ut.ee, jessica.lindstrom@helsinki.fi, m.dailidenaite@gmail.com Abstract In text processing, deep neural networks mostly use word embeddings as an input. Embeddings have to ensure that relations between words are reflected through distances in a high-dimensional numeric space. To compare the quality of different text embeddings, typically, we use benchmark datasets. We present a collection of such datasets for the word analogy task in nine languages: Croatian, English, Estonian, Finnish, Latvian, Lithuanian, Russian, Slovenian, and Swedish. We designed the monolingual analogy task to be much more culturally independent and also constructed cross-lingual analogy datasets for the involved languages. We present basic statistics of the created datasets and their initial evaluation using fastText embeddings. Keywords: word embeddings, analogy task, evaluation, less-resourced languages 1. Introduction As an input, neural networks require numerical data. Text embeddings provide such an input, ensuring that relations between words are reflected in the distances and directions in high-dimensional numeric space. There are many dis- tinct models producing embedding vectors, using different specialized learning tasks, e.g., word2vec (Mikolov et al., 2013b), GloVe (Pennington et al., 2014), and fastText (Bo- janowski et al., 2017). For training, the embeddings algo- rithms use large monolingual corpora. To compare the quality of different text embeddings, typi- cally we use benchmark datasets. In this work, we present a collection of such datasets for the word analogy task in nine languages: Croatian, English, Estonian, Finnish, Latvian, Lithuanian, Russian, Slovenian, and Swedish. To make the datasets sensible for all languages, we designed the analogy task to be culturally neutral. Specifically, we avoided anal- ogy categories that exhibit single culture or single country examples, such as NFL teams, US cities, or US states. The word analogy task was popularized by Mikolov et al. (2013c). The goal is to find a term y for a given term x so that the relationship between x and y best resembles the given relationship a : b. There are two main groups of cat- egories: semantic and syntactic. To illustrate a semantic relationship, consider for example that the word pair a : b is given as “Finland : Helsinki”. The task is to find the term y corresponding to the relationship “Sweden : y”, with the expected answer being y = Stockholm. In syntactic cat- egories, each category refers to a grammatical feature, for example adjective degrees of comparison. The two words in any given pair then have a common stem (or even the same lemma), for example, given the word pair “long : longer”, we …

Content truncated to 500 words