BERTifying Sinhala - A Comprehensive Analysis of Pre-trained Language Models for Sinhala Text Classification

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7377–7385 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 7377 BERTifying Sinhala - A Comprehensive Analysis of Pre-trained Language Models for Sinhala Text Classification Vinura Dhananjaya, Piyumal Demotte, Surangika Ranathunga, Sanath Jayasena Department of Computer Science and Engineering University of Moratuwa, Katubedda 10400, Sri Lanka dhananjayagv.21@uom.lk, piyumalanthony.16@cse.mrt.ac.lk, surangika@cse.mrt.ac.lk, sanath@uom.lk Abstract This research provides the first comprehensive analysis of the performance of pre-trained language models for Sinhala text classification. We test on a set of different Sinhala text classification tasks and our analysis shows that out of the pre-trained multilingual models that include Sinhala (XLM-R, LaBSE, and LASER), XLM-R is the best model by far for Sinhala text classification. We also pre-train two RoBERTa-based monolingual Sinhala models, which are far superior to the existing pre-trained language models for Sinhala. We show that when fine-tuned, these pre-trained language models set a very strong baseline for Sinhala text classification and are robust in situations where labeled data is insufficient for fine-tuning. We further provide a set of recommendations for using pre-trained models for Sinhala text classification. We also introduce new annotated datasets useful for future research in Sinhala text classification and publicly release our pre-trained models. Keywords: Language Models, Monolingual, Multilingual, Text Classification 1. Introduction Large-scale monolingual pre-trained language models (MonoLMs) such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and their multilingual descendants mBERT (Devlin et al., 2019) and XLM- R (Conneau et al., 2019) ( respectively), have shown promising results for high-resource as well as low- resource languages, particularly for text classifica- tion (Wu and Dredze, 2019; Aguilar et al., 2020). Following this early success, many empirical studies have been carried out to determine the performance of these models on different settings. However, most of these studies focused on a limited number of lan- guages. Moreover, experimental results show that the performance of the pre-trained Multilingual Language Models (MultiLMs) depends on many factors - such as the amount of language data used during the pre- training, the relatedness of a language to the other lan- guages in the pre-trained model, language characteris- tics, the typography of the language, and the amount of fine-tuning data used (Wu and Dredze, 2020; Dod- dapaneni et al., 2021). Thus, the results reported in these studies cannot be generalized across languages. For MonoLMs, a major deciding factor is the amount of monolingual data used in the model pre-training stage (Rust et al., 2020). As noted by Soria et al. (2018), ‘a Natural Lan- guage Processing (NLP) system can be measured only in terms of its usefulness for the end-users’. In other words, the usefulness of pre-trained language models on a language depends on their ability to provide ac- ceptable results for the NLP tasks of the considered language, but not by their performance on some other set of languages. Thus, it is imperative that we carry out extensive evaluation of the pre-trained models for the …

Content truncated to 500 words