Improving Large-scale Language Models and Resources for Filipino

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6548–6555 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6548 Improving Large-scale Language Models and Resources for Filipino Jan Christian Blaise Cruz and Charibeth Cheng Center for Language Technologies (CeLT), De La Salle University, Manila 2401 Taft Ave., Malate, Manila, Philippines {jan christian cruz,charibeth.cheng}@dlsu.edu.ph Abstract In this paper, we improve on existing language resources for the low-resource Filipino language in two ways. First, we outline the construction of the TLUnified dataset, a large-scale pretraining corpus that serves as an improvement over smaller existing pretraining datasets for the language in terms of scale and topic variety. Second, we pretrain new Transformer language models following the RoBERTa pretraining technique to supplant existing models trained with small corpora. Our new RoBERTa models show significant improvements over existing Filipino models in three benchmark datasets with an average gain of 4.47% test accuracy across three classification tasks with varying difficulty. Keywords: Transformers, Corpus Creation, Benchmarking 1. Introduction Unlike High-resource Languages (HRL) such as En- glish, German, and French, Low-resource Languages (LRL) suffer from a lack of benchmark datasets, databases, linguistic tools, and pretrained models that impede the progress of research within those lan- guages. Despite the growing success of methods that intrinsi- cally learn from little data (Deng et al., 2020; Lee et al., 2021), creating “more data” remains a very signif- icant fundamental task in NLP. Given the data-hungry nature of the neural networks that are prevalent in NLP today, creating new datasets to train from is the most ef- ficient way to improve model performance. In addition, cleverly-constructed datasets also reveal new insights into the models we commonly use, letting us gauge their true performance and expose hidden weaknesses (Maudslay and Cotterell, 2021). In this paper, we improve upon the existing resources for Filipino, a low-resource language spoken in the Philippines. We create a larger, more topically-varied large-scale pretraining dataset that improves upon the existing WikiText-TL-39 (Cruz and Cheng, 2019) that is too small and too topically-narrow to create robust models that perform well in modern NLP. We also pro- duce new RoBERTa pretrained models using our pre- training dataset that supplant existing models trained with less data (Cruz and Cheng, 2020). 2. Resource Creation In this section, we outline our full methodology for re- source creation. First, we introduce the construction of our new large-scale pretraining dataset. Next, we detail the pretraining steps for our new RoBERTa mod- els. Lastly, we introduce the task datasets that we use to benchmark performance for our new pretrained mod- els. 2.1. The TLUnified Dataset To effectively pretrain a large transformer for down- stream tasks, we require an equally large pretraining corpus of high-quality Filipino text. We construct our pretraining corpus by combining a number of available Filipino corpora, including: • Bilingual Text Data – Bitext datasets are used for training Machine Translation models and contain crawled and aligned data from multiple sources. We collected multiple bitexts, extracted …

Content truncated to 500 words