Subword Evenness (SuE) as a Predictor of Cross-lingual Transfer to Low-resource Languages

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7428 - 7445 December 7-11, 2022 ©2022 Association for Computational Linguistics Subword Evenness (SuE) as a Predictor of Cross-lingual Transfer to Low-resource Languages Olga Pelloni1 Anastassia Shaitarova2 Tanja Samardzic1 1Text Group, URPP Language and Space, 2Department of Computational Linguistics, University of Zurich, Switzerland olga.pelloni@gmail.com, {anastassia.shaitarova, tanja.samardzic}@uzh.ch Abstract Pre-trained multilingual models, such as mBERT, XLM-R and mT5, are used to im- prove the performance on various tasks in low- resource languages via cross-lingual transfer. In this framework, English is usually seen as the most natural choice for a transfer lan- guage (for fine-tuning or continued training of a multilingual pre-trained model), but it has been revealed recently that this is often not the best choice. The success of cross-lingual transfer seems to depend on some properties of languages, which are currently hard to ex- plain. Successful transfer often happens be- tween unrelated languages and it often can- not be explained by data-dependent factors. In this study, we show that languages written in non-Latin and non-alphabetic scripts (mostly Asian languages) are the best choices for im- proving performance on the task of Masked Language Modelling (MLM) in a diverse set of 30 low-resource languages and that the success of the transfer is well predicted by our novel measure of Subword Evenness (SuE). Transfer- ring language models over the languages that score low on our measure results in the low- est average perplexity over target low-resource languages. Our correlation coefficients ob- tained with three different pre-trained multilin- gual models are consistently higher than all the other predictors, including text-based measures (type-token ratio, entropy) and linguistically motivated choice (genealogical and typological proximity). 1 Introduction Since pre-trained multilingual models became available, the most common approach to NLP tasks on low-resource languages has been cross-lingual transfer learning. After initial tests on cross-lingual ability within the languages covered by the multi- lingual models (Pires et al., 2019; Wu and Dredze, 2020; Libovický et al., 2020), a more recent line of research shows that new languages (not seen in Figure 1: Best cross-lingual transfer results mentioned in previous work. The size of each circle is proportional to the number of mentions of a given transfer-target pair. English is the best transfer language for European target languages. For target languages outside of Europe, the best transfer languages are hard to predict. the training set) are best processed by continued training or fine-tuning (depending on the task) of a pre-trained multilingual model on one transfer language, even if the transfer language is differ- ent from the target language. As a matter of fact, a single transfer language can help improve the performance on many target languages (Lin et al., 2019; Lauscher et al., 2020; Turc et al., 2021). This insight is especially important for low-resource lan- guages, for which only test data is typically avail- able. Multilingual models are usually fine-tuned or (additionally) trained on a high-resource language and the resulting weights are applied as zero- or …

Content truncated to 500 words