Evaluation of Croatian Word Embeddings

Evaluation of Croatian Word Embeddings Luk´aˇs Svoboda1, Slobodan Beliga2 1) Department of Computer Science and Engineering, University of West Bohemia Univerzitn´ı 22, 306 14 Plzeˇn, Czech Republic 2) Department of Informatics, University of Rijeka Radmile Matejˇci´c 2, 51000 Rijeka, Croatia svobikl@kiv.zcu.cz1, sbeliga@inf.uniri.hr2 Abstract Croatian is poorly resourced and highly inflected language from Slavic language family. Nowadays, research is focusing mostly on English. We created a new word analogy dataset based on the original English Word2vec word analogy dataset and added some of the specific linguistic aspects from the Croatian language. Next, we created Croatian WordSim353 and RG65 datasets for a basic evaluation of word similarities. We compared created datasets on two popular word representation models, based on Word2Vec tool and fastText tool. Models have been trained on 1.37B tokens training data corpus and tested on a new robust Croatian word analogy dataset. Results show that models are able to create meaningful word representation. This research has shown that free word order and the higher morphological complexity of Croatian language influences the quality of resulting word embeddings. Keywords: Croatian word embeddings, Croatian word analogy, Croatian language, Slavic language family, Word2Vec, FastText, Croatian word similarity dataset, WordSim353, RG65 1. Introduction Word2Vec and FastText are tools that create models rep- resenting words as vectors of real numbers from high- dimensional space. Word representations are based on Dis- tributional Hypothesis (Harris, 1954), where the context for each word is given by its nearby words. The goal of such representations is to capture the syntactic and semantic re- lationship between words. It was shown that the word vectors can be successfully used in order to improve and/or simplify many NLP applications (Collobert and Weston, 2008; Collobert et al., 2011). There are also NLP tasks, where word embeddings does not help much (Andreas and Klein, 2014). Most of the work is focused on English. Recently the com- munity has realized that the research should focus on other languages with rich morphology and different syntax (Be- rardi et al., 2015; Elrazzaz et al., 2017; K¨oper et al., 2015; Svoboda and Brychc´ın, 2016), but there is still a little atten- tion to languages from Slavic family. These languages are highly inflected and have a relatively free word order. Since there are open questions related to the embeddings in the Slavic language family, in this paper, we will focus mainly on Croatian word embeddings, from the South Slavic lan- guage family. With the aim of expanding existing findings about Croatian word embeddings, in this paper we will: 1. Compare different word embeddings methods on Croatian language which is not deeply explored, and according to its features, belongs to highly inflected language (words can have seven different cases for sin- gular and seven for plural, genders, and numbers). 2. For the purposes of the word embeddings experi- ments, we will create three new datasets. Two ba- sic word similarity datasets based on original Word- Sim353 (Finkelstein et al., 2002) and RG65 (Ruben- stein and Goodenough, 1965) will be translated to …

Content truncated to 500 words