Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning

Proceedings of the 29th International Conference on Computational Linguistics, pages 4336–4349 October 12–17, 2022. 4336 Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning Jesujoba O. Alabi∗, David Ifeoluwa Adelani∗, Marius Mosbach, and Dietrich Klakow Spoken Language Systems (LSV), Saarland University, Saarland Informatics Campus, Germany {jalabi,didelani,mmosbach,dklakow}@lsv.uni-saarland.de Abstract Multilingual pre-trained language models (PLMs) have demonstrated impressive per- formance on several downstream tasks for both high-resourced and low-resourced lan- guages. However, there is still a large perfor- mance drop for languages unseen during pre- training, especially African languages. One of the most effective approaches to adapt to a new language is language adaptive fine-tuning (LAFT) — fine-tuning a multilingual PLM on monolingual texts of a language using the pre-training objective. However, adapting to a target language individually takes a large disk space and limits the cross-lingual transfer abilities of the resulting models because they have been specialized for a single language. In this paper, we perform multilingual adap- tive fine-tuning on 17 most-resourced African languages and three other high-resource lan- guages widely spoken on the African conti- nent to encourage cross-lingual transfer learn- ing. To further specialize the multilingual PLM, we removed vocabulary tokens from the embedding layer that corresponds to non- African writing scripts before MAFT, thus re- ducing the model size by around 50%. Our evaluation on two multilingual PLMs (AfriB- ERTa and XLM-R) and three NLP tasks (NER, news topic classification, and sentiment clas- sification) shows that our approach is com- petitive to applying LAFT on individual lan- guages while requiring significantly less disk space. Additionally, we show that our adapted PLM also improves the zero-shot cross-lingual transfer abilities of parameter efficient fine- tuning methods. 1 Introduction Recent advances in the development of multilingual pre-trained language models (PLMs) like mBERT (De- vlin et al., 2019), XLM-R (Conneau et al., 2020), and RemBERT (Chung et al., 2021) have led to significant performance gains on a wide range of cross-lingual ∗* Equal contribution. transfer tasks. Due to the curse of multilinguality (Con- neau et al., 2020) — a trade-off between language cov- erage and model capacity — and non-availability of pre-training corpora for many low-resource languages, multilingual PLMs are often trained on about 100 lan- guages. Despite the limitations of language cover- age, multilingual PLMs have been shown to transfer to several low-resource languages unseen during pre- training. Although, there is still a large performance gap compared to languages seen during pre-training. One of the most effective approaches to adapt to a new language is language adaptive fine-tuning (LAFT) — fine-tuning a multilingual PLM on monolingual texts in the target language using the same pre-training objective. This has been shown to lead to big gains on many cross-lingual transfer tasks (Pfeiffer et al., 2020), and low-resource languages (Muller et al., 2021; Chau & Smith, 2021), including African languages (Alabi et al., 2020; Adelani et al., 2021). Nevertheless, adapt- ing a model to each target language individually takes large disk space, and limits the cross-lingual transfer abilities of the resulting …

Content truncated to 500 words