LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5080–5089 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5080 LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish Cedric Lothritz1, Bertrand Lebichot1, Kevin Allix1, Lisa Veiber1, Tegawend´e F. Bissyand´e1, Jacques Klein1, Andrey Boytsov2, Anne Goujon2, Cl´ement Lefebvre2 1University of Luxembourg 16, rue Richard Coudenhove-Kalergi, L-1359 Luxembourg 2 Banque BGL BNP Paribas 2 50, avenue J.F Kennedy, L-2951 Luxembourg {cedric.lothritz, bertrand.lebichot, kevin.allix, lisa.veiber, tegawende.bissyande, jacques.klein}@uni.lu {andrey.boytsov, anne.goujon, clement.c.lefebvre}@bgl.lu Abstract Pre-trained Language Models such as BERT have become ubiquitous in NLP where they have achieved state-of-the-art performance in most NLP tasks. While these models are readily available for English and other widely spoken languages, they remain scarce for low-resource languages such as Luxembourgish. In this paper, we present LuxemBERT, a BERT model for the Luxembourgish language that we create using the following approach: we augment the pre-training dataset by considering text data from a closely related language that we partially translate using a simple and straightforward method. We are then able to produce the LuxemBERT model, which we show to be effective for various NLP tasks: it outperforms a simple baseline built with the available Luxembourgish text data as well the multilingual mBERT model, which is currently the only option for transformer-based language models in Luxembourgish. Furthermore, we present datasets for various downstream NLP tasks that we created for this study and will make available to researchers on request. Keywords: Language Models, Less-Resourced Languages, NLP Datasets 1. Introduction Pre-trained Language Models for NLP tasks have be- come increasingly popular over the last years and will likely continue to thrive in the years to come. Their usefulness is immediately obvious as they mitigate the need to train specific NLP models from scratch and can be reused for multiple tasks through fine-tuning. In par- ticular, BERT (Devlin et al., 2018) and its variants such as RoBERTa (Liu et al., 2019), DistilBERT (Sanh et al., 2019), and XLNet (Yang et al., 2019) are widely leveraged by researchers and practitioners alike. Un- fortunately, while these models generally reach state- of-the-art performances for most downstream tasks, they present a significant caveat as the pre-training step requires huge amounts of computing resources, time, and, most importantly, data. For example, the origi- nal BERT model, which targeted the English language, was trained on the entire English version of Wikipedia as well as the BooksCorpus (Zhu et al., 2015), amount- ing to a 3.3 billion words dataset. While this amount of data is readily available for widely spoken languages such as English, German and French, it is not the case for many low-resource languages such as Luxembour- gish. This data scarcity therefore becomes a major ob- stacle for building adequate language models. Data from low-resource languages have been included along many other languages to build a multilingual BERT (mBERT) which researchers and practitioners resort to for dealing with NLP tasks. Unfortunately, al- though mBERT-based models generally perform well, …

Content truncated to 500 words