High Quality ELMo Embeddings for Seven Less-Resourced Languages

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4731–4738 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4731 High Quality ELMo Embeddings for Seven Less-Resourced Languages Matej Ulˇcar, Marko Robnik-ˇSikonja University of Ljubljana, Faculty of Computer and Information Science Veˇcna pot 113, SI-1000 Ljubljana, Slovenia {matej.ulcar, marko.robnik}@fri.uni-lj.si Abstract Recent results show that deep neural networks using contextual embeddings significantly outperform non-contextual embeddings on a majority of text classification tasks. We offer precomputed embeddings from popular contextual ELMo model for seven languages: Croatian, Estonian, Finnish, Latvian, Lithuanian, Slovenian, and Swedish. We demonstrate that the quality of embeddings strongly depends on the size of the training set and show that existing publicly available ELMo embeddings for listed languages shall be improved. We train new ELMo embeddings on much larger training sets and show their advantage over baseline non-contextual fastText embeddings. In evaluation, we use two benchmarks, the analogy task and the NER task. Keywords: word embeddings, contextual embeddings, ELMo, less-resourced languages, analogy task, named entity recognition 1. Introduction Word embeddings are representations of words in numer- ical form, as vectors of typically several hundred dimen- sions. The vectors are used as an input to machine learn- ing models; for complex language processing tasks these are typically deep neural networks. The embedding vec- tors are obtained from specialized learning tasks, based on neural networks, e.g., word2vec (Mikolov et al., 2013b), GloVe (Pennington et al., 2014), fastText (Bojanowski et al., 2017), ELMo (Peters et al., 2018), and BERT (Devlin et al., 2019). For training, the embeddings algorithms use large monolingual corpora that encode important informa- tion about word meaning as distances between vectors. In order to enable downstream machine learning on text un- derstanding tasks, the embeddings shall preserve semantic relations between words, and this is true even across lan- guages. Probably the best known word embeddings are produced by the word2vec method (Mikolov et al., 2013c). The problem with word2vec embeddings is their failure to express poly- semous words. During training of an embedding, all senses of a given word (e.g., paper as a material, as a newspa- per, as a scientific work, and as an exam) contribute rel- evant information in proportion to their frequency in the training corpus. This causes the final vector to be placed somewhere in the weighted middle of all words’ meanings. Consequently, rare meanings of words are poorly expressed with word2vec and the resulting vectors do not offer good semantic representations. For example, none of the 50 clos- est vectors of the word paper is related to science1. The idea of contextual embeddings is to generate a dif- ferent vector for each context a word appears in and the context is typically defined sentence-wise. To a large ex- tent, this solves the problems with word polysemy, i.e. the context of a sentence is typically enough to disambiguate different meanings of a word for humans and so it is for the 1This can be checked with a demo …

Content truncated to 500 words