Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measures

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6461–6470 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6461 Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measures Jonathan Dunn, Haipeng Li, Damian Sastre University of Canterbury, Department of Linguistics and the New Zealand Institute for Language, Brain and Behaviour Christchurch, New Zealand jonathan.dunn@canterbury.ac.nz, haipeng.li@canterbury.ac.nz, dsa105@uclive.ac.nz Abstract This paper simulates a low-resource setting across 17 languages in order to evaluate embedding similarity, stability, and reliability under different conditions. The goal is to use corpus similarity measures before training to predict properties of embeddings after training. The main contribution of the paper is to show that it is possible to predict downstream embedding similarity using upstream corpus similarity measures. This finding is then applied to low-resource settings by modelling the reliability of embeddings created from very limited training data. Results show that it is possible to estimate the reliability of low-resource embeddings using corpus similarity measures that remain robust on small amounts of data. These findings have significant implications for the evaluation of truly low-resource languages in which such systematic downstream validation methods are not possible because of data limitations. Keywords: low-resource languages, embedding similarity, corpus similarity, register variation 1. Validating Low-Resource Embeddings This paper simulates a low-resource setting for 17 non- English languages in order to evaluate embedding simi- larity, stability, and reliability under different conditions. While these are actually high-resource languages, we are able to simulate a low-resource setting across dif- ferent language families, writing systems, and types of morphology by constraining both the amount and the type of data that is made available. We then try to pre- dict the downstream similarity, stability, and reliability of embeddings given upstream properties of the train- ing corpora. The larger goal is to predict the reliability of embeddings in truly low-resource settings in which such experiments are not possible. The key finding of the paper is that there is a strong relationship between the similarity of training corpora and the similarity of embeddings, a relationship that extends across a diverse range of languages and registers. Embeddings remain a key representation within NLP, as shown by many samples of recent work (Miaschi and Dell’Orletta, 2020; Adelmann et al., 2021). At the same time, however, recent work has also shown that embeddings are surprisingly variable (Wendlandt et al., 2018; Burdick et al., 2021). Such work has shown that high-resource languages like English, with many bil- lions of words available for training, have embeddings that differ by data set (Antoniak and Mimno, 2018), by geographic population (Dunn and Adams, 2020), and even by random iterations on the same data set (Hellrich et al., 2019). The basic implication is that, for instance, English embeddings from web data from South Asia are expected to be quite different from English embeddings from American news articles. For high-resource languages, such variability is miti- gated by the wide availability of in-domain training data for most tasks. But for low-resource …

Content truncated to 500 words