BasqueGLUE: A Natural Language Understanding Benchmark for Basque

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1603–1612 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1603 BasqueGLUE: A Natural Language Understanding Benchmark for Basque Gorka Urbizu1, I˜naki San Vicente1, Xabier Saralegi1, Rodrigo Agerri2, Aitor Soroa2 1Elhuyar Foundation 2HiTZ Center - Ixa, University of the Basque Country UPV/EHU {g.urbizu, i.sanvicente, x.saralegi}@elhuyar.eus {rodrigo.agerri, a.soroa}@ehu.eus Abstract Natural Language Understanding (NLU) technology has improved significantly over the last few years and multitask bench- marks such as GLUE are key to evaluate this improvement in a robust and general way. These benchmarks take into account a wide and diverse set of NLU tasks that require some form of language understanding, beyond the detection of superficial, textual clues. However, they are costly to develop and language-dependent, and therefore they are only available for a small number of languages. In this paper, we present BasqueGLUE, the first NLU benchmark for Basque, a less-resourced language, which has been elaborated from previously existing datasets and following similar criteria to those used for the construction of GLUE and SuperGLUE. We also report the evaluation of two state-of-the-art language models for Basque on BasqueGLUE, thus providing a strong baseline to compare upon. BasqueGLUE is freely available under an open license. Keywords: Neural Language Models, Natural Language Understanding, Less-Resourced Languages 1. Introduction The Transformer architecture (Vaswani et al., 2017) has led to a family of powerful neural language mod- els (LM) that can scale up to billions of parameters, and which are pre-trained using enormous quantities of text (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019). These models have achieved remarkable results in many Natural Language Understanding (NLU) tasks by following the transfer learning approach, where the LMs are first pre-trained over vast amounts of unanno- tated text, and then fine-tuned to a certain downstream task using a comparatively much smaller amount of an- notated data. Experimentation and evaluation with neural language models in NLU have been greatly facilitated by the ex- istence of evaluation frameworks such as GLUE (Wang et al., 2018) or SuperGLUE (Wang et al., 2019), that assess the capacity of the models to understand text be- yond the detection of superficial clues. Those bench- marks provide a unified and robust evaluation frame- work for evaluating NLU systems that is not exclusive to a single task, genre, or dataset. Initially, most NLU assessment frameworks were focused on English, but gradually such resources are being published for other languages (Xu et al., 2020; Le et al., 2020; Wilie et al., 2020; Kakwani et al., 2020; Shavrina et al., 2020; Park et al., 2021). This paper presents BasqueGLUE1, the first NLU evaluation framework for Basque. We believe that BasqueGLUE is a significant contribution towards de- veloping NLU tools in Basque, which we believe will 1https://github.com/Elhuyar/BasqueGLUE facilitate the technological advance for the Basque language. In order to create BasqueGLUE we took as a reference the GLUE and SuperGLUE frame- works. When possible, we re-used existing datasets …

Content truncated to 500 words