Evaluating Word Embeddings in Extremely Under-Resourced Languages: A Case Study in Bribri

Proceedings of the 29th International Conference on Computational Linguistics, pages 4455–4467 October 12–17, 2022. 4455 Evaluating Word Embeddings in Extremely Under-Resourced Languages: A Case Study in Bribri Rolando Coto-Solano Dartmouth College rolando.a.coto.solano@dartmouth.edu Abstract Word embeddings are critical for numerous NLP tasks but their evaluation in actual under- resourced settings needs further examination. This paper presents a case study in Bribri, a Chibchan language from Costa Rica. Four ex- periments were adapted from English: Word similarities, WordSim353 correlations, odd- one-out tasks and analogies. Here we discuss their adaptation to an under-resourced Indige- nous language and we use them to measure se- mantic and morphological learning. We trained 96 word2vec models with different hyperpa- rameter combinations. The best models for this under-resourced scenario were Skip-grams with an intermediate size (100 dimensions) and large window sizes (10). These had an aver- age correlation of r=0.28 with WordSim353, a 76% accuracy in semantic odd-one-out and 70% accuracy in structural/morphological odd- one-out. The performance was lower for the analogies: The best models could find the ap- propriate semantic target amongst the first 25 results approximately 60% of the times, but could only find the morphological/structural target 11% of the times. Future research needs to further explore the patterns of morphologi- cal/structural learning, to examine the behavior of deep learning embeddings, and to establish a human baseline. This project seeks to improve Bribri NLP and ultimately help in its mainte- nance and revitalization. 1 Introduction Word embeddings are dense vectors that describe the semantics of words (Landauer et al., 1997). They are calculated by collecting the words around a specific word and using them to create a numer- ical vector that can determine semantic similarity. For example, the words spinach and kale share neighboring words like garlic and cooked, which would make them similar to each other and differ- ent from words with other neighbors, such as ham- mer. Embeddings can vary in size, or dimensional- ity: Word embeddings from systems like Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) can range from size 10 to 300, while the em- beddings from GPT-3 (Brown et al., 2020) can have up to 12288 dimensions. Embeddings are an integral part of a number of NLP tasks. Because these vectors can be used to calculate cosine distance, one could calculate the distance between words like king and queen, and examine if this is similar to the distance between man and woman. Learning these patterns entails that the system is learning the gender distinctions in the words and that it is generalizing patterns across words. This would make an embedding a type of language model which could be enlisted to solve tasks like finding the “odd” word in a triad of words. For example, it could determine that tiger is the odd word in the set {man, woman, tiger} given its greater cosine distance from the other two. This is known as the odd-one-out task. Embeddings can also solve analogies by using vector algebra. The embedding vectors …

Content truncated to 500 words