Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection

Findings of the Association for Computational Linguistics: EACL 2023, pages 1251–1266 May 2-6, 2023 ©2023 Association for Computational Linguistics Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection Tolúlo.pé. Ògúnrè.mí Stanford University tolulope@cs.stanford.edu Dan Jurafsky Stanford University jurafsky@stanford.edu Christopher D. Manning Stanford University manning@cs.stanford.edu Abstract With the prominence of large pretrained lan- guage models, low-resource languages are rarely modelled monolingually and become victims of the “curse of multilinguality” in massively multilingual models. Recently, Afri- BERTa showed that training transformer mod- els from scratch on 1GB of data from many unrelated African languages outperforms mas- sively multilingual models on downstream NLP tasks. Here we extend this direction, focus- ing on the use of related languages. We pro- pose that training on smaller amounts of data but from related languages could match the performance of models trained on large, un- related data. We test our hypothesis on the Niger-Congo family and its Bantu and Volta- Niger sub-families, pretraining models with data solely from Niger-Congo languages and finetuning on 4 downstream tasks: NER, part- of-speech tagging, sentiment analysis and text classification. We find that models trained on genetically related languages achieve equal per- formance on downstream tasks in low-resource languages despite using less training data. We recommend selecting training data based on language-relatedness when pretraining lan- guage models for low-resource languages. 1 Introduction Since the introduction of the large pretrained lan- guage models (Devlin et al., 2019), low-resource languages have not had the opportunity to be treated in the same way as high-resource languages such as English, French or Mandarin Chinese. Mas- sively multilingual models trained using a mix- ture of high and low-resource languages such as mBERT (Devlin et al., 2019), XLM-RoBERTa (Conneau et al., 2020) or mT5, (Xue et al., 2021) have been proposed as a solution. Yet these do not work as well on low-resource languages as they do on high-resource languages due to the “curse of multilinguality” (Conneau et al., 2020), where an increase of languages in a model leads to capac- ity dilution, negatively affecting performance for all languages. This makes massively multilingual models sub-optimal solutions for such languages. The quality of the training data for low-resource languages seems to differ greatly to that of high- resource languages (Kreutzer et al., 2022). The AfriBERTa models (Ogueji et al., 2021) demon- strate the considerable success of pretrained repre- sentations when trained with a ‘small’ (1GB), high- quality dataset focused on eleven languages of a single continent – Africa. AfriBERTa Large out- performs the larger, massively multilingual models on named-entity-recognition (NER) and text clas- sification for various African languages. While this continental approach is promising, it uses a mixture of different language families that are not genetically related. Here, we propose using language relatedness in lieu of general geographic proximity of languages to pretrain transformer models. We test this hy- pothesis by grouping training data by language family and then testing on four tasks: NER, Part-of- Speech Tagging (POS Tagging), Sentiment Anal- ysis and Text Classification. New models trained …

Content truncated to 500 words