How to Adapt Your Pretrained Multilingual Model to 1600 Languages

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4555–4567 August 1–6, 2021. ©2021 Association for Computational Linguistics 4555 How to Adapt Your Pretrained Multilingual Model to 1600 Languages Abteen Ebrahimi and Katharina Kann University of Colorado Boulder {abteen.ebrahimi,katharina.kann}@colorado.edu Abstract Pretrained multilingual models (PMMs) en- able zero-shot learning via cross-lingual trans- fer, performing best for languages seen dur- ing pretraining. While methods exist to im- prove performance for unseen languages, they have almost exclusively been evaluated using amounts of raw text only available for a small fraction of the world’s languages. In this paper, we evaluate the performance of existing meth- ods to adapt PMMs to new languages using a resource available for over 1600 languages: the New Testament. This is challenging for two reasons: (1) the small corpus size, and (2) the narrow domain. While performance drops for all approaches, we surprisingly still see gains of up to 17.69% accuracy for part-of- speech tagging and 6.29 F1 for NER on aver- age over all languages as compared to XLM-R. Another unexpected finding is that continued pretraining, the simplest approach, performs best. Finally, we perform a case study to dis- entangle the effects of domain and size and to shed light on the influence of the finetuning source language. 1 Introduction Pretrained multilingual models (PMMs) are a straightforward way to enable zero-shot learning via cross-lingual transfer, thus eliminating the need for labeled data for the target task and language. However, downstream performance is highest for languages that are well represented in the pretrain- ing data or linguistically similar to a well repre- sented language. Performance degrades as repre- sentation decreases, with languages not seen during pretraining generally having the worst performance. In the most extreme case, when a language’s script is completely unknown to the model, zero-shot per- formance is effectively random. While multiple methods have been shown to improve the performance of transfer to underrep- Figure 1: The number of space-separated words in the Bible and Wikipedia for six low-resource languages used in our experiments; plotted on a log scale. resented languages (cf. Section 2.3), previous work has evaluated them using unlabeled data from sources available for a relatively small number of languages, such as Wikipedia or Common Crawl, which cover 3161 and 1602 languages, respectively. Due to this low coverage, the languages that would most benefit from these methods are precisely those which do not have the necessary amounts of mono- lingual data to implement them as-is. To enable the use of PMMs for truly low-resource languages, where they can, e.g., assist language documentation or revitalization, it is important to understand how state-of-the-art adaptation methods act in a setting more broadly applicable to many languages. In this paper, we ask the following question: Can we use the Bible – a resource available for roughly 1600 languages – to improve a PMM’s zero-shot performance on an unseen target language? And, if so, what adaptation method …

Content truncated to 500 words