gaBERT — an Irish Language Model

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4774–4788 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4774 gaBERT — an Irish Language Model James Barry1, Joachim Wagner2, Lauren Cassidy1 Alan Cowap3, Teresa Lynn1, Abigail Walsh1 Mícheál J. Ó Meachair4, Jennifer Foster2 1,2,3School of Computing, Dublin City University, 1ADAPT Centre 3SFI Centre for Research Training in Machine Learning at Dublin City University 4Fiontar & Scoil na Gaeilge 1 {firstname.lastname}@adaptcentre.ie 2 {firstname.lastname}@dcu.ie 3 alan.cowap2@mail.dcu.ie 4 micheal.omeachair@dcu.ie Abstract The BERT family of neural language models have become highly popular due to their ability to provide sequences of text with rich context-sensitive token encodings which are able to generalise well to many NLP tasks. We introduce gaBERT, a monolingual BERT model for the Irish language. We compare our gaBERT model to multilingual BERT and the monolingual Irish WikiBERT, and we show that gaBERT provides better representations for a downstream parsing task. We also show how different filtering criteria, vocabulary size and the choice of subword tokenisation model affect downstream performance. We compare the results of fine-tuning a gaBERT model with an mBERT model for the task of identifying verbal multiword expressions, and show that the fine-tuned gaBERT model also performs better at this task. We release gaBERT and related code to the community. Keywords: BERT, Irish 1. Introduction The technique of fine-tuning a self-supervised language model has become ubiquitous in Natural Language Pro- cessing (NLP) because models trained in this way have advanced evaluation scores on many tasks (Radford et al., 2018; Peters et al., 2018; Devlin et al., 2019). Ar- guably the most popular architecture is BERT (Devlin et al., 2019) which uses stacks of transformer blocks to predict the identity of a masked token and to pre- dict whether two sequences are contiguous. It has spawned many variants (Liu et al., 2019; Lan et al., 2019) and much analysis (Jawahar et al., 2019; Chi et al., 2020; Rogers et al., 2020). In this paper, we introduce gaBERT, a monolingual model of Irish. Although Irish is the first official language of the Repub- lic of Ireland, only a minority, 1.5% of the population (CSO, 2016), use it in their everyday lives outside of the education system. As the less dominant language in a bilingual community, the availability of Irish language technology is important since it facilitates Irish speakers and learners to continue to use the language in their in- creasingly digital daily lives. In terms of technological support however, Irish is a low-resourced language and significantly lacking in speech and language tools and resources (Lynn, 2022). From a linguistic perspective, the Irish language is an inflected language, sharing linguistic features with other Celtic languages such as verb-subject-object (VSO) word order, initial mutation (lenition and eclipsis) and inflected prepositions. Inflection is common through suffixation, marking tense, number and person, while nouns are inflected for number and case. Nouns are either masculine or feminine in grammatical gender, which in turn influences declension-dependent inflec- tions. …

Content truncated to 500 words