bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8733–8759 July 9-14, 2023 ©2023 Association for Computational Linguistics bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark Momchil Hardalov∗ AWS AI Labs Pepa Atanasova University of Copenhagen, DIKU Todor Mihaylov Meta AI† Galia Angelova Kiril Simov Petya Osenova IICT, Bulgarian Academy of Sciences Ves Stoyanov Meta AI Ivan Koychev Sofia University Preslav Nakov MBZUAI Dragomir Radev Yale University Abstract We present bgGLUE (Bulgarian General Lan- guage Understanding Evaluation), a benchmark for evaluating language models on Natural Lan- guage Understanding (NLU) tasks in Bulgarian. Our benchmark includes NLU tasks targeting a variety of NLP problems (e.g., natural lan- guage inference, fact-checking, named entity recognition, sentiment analysis, question an- swering, etc.) and machine learning tasks (se- quence labeling, document-level classification, and regression). We run the first systematic evaluation of pre-trained language models for Bulgarian, comparing and contrasting results across the nine tasks in the benchmark. The evaluation results show strong performance on sequence labeling tasks, but there is a lot of room for improvement for tasks that require more complex reasoning. We make bgGLUE publicly available together with the fine-tuning and the evaluation code, as well as a public leaderboard at https://bgglue.github.io, and we hope that it will enable further advancements in developing NLU models for Bulgarian. 1 Introduction Natural Language Understanding (NLU) bench- marks, such as GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a), were designed for a rigorous evaluation of language models on a diverse set of natural language understanding (NLU) tasks. The wide adoption of such bench- marks has driven the rapid development of models that perform well on the tasks that are part of these benchmarks, but also beyond (Devlin et al., 2019; Liu et al., 2019). However, until recently, the focus of such benchmarks has been on English, with lit- tle interest in other languages (Bender, 2011; Ponti et al., 2019). ∗Work done while Momchil was in the Sofia University, prior to joining Amazon. †Meta AI is not involved in the creation, release, or hosting of the datasets in the benchmark. To address this, recent work has designed bench- marks to test models on NLU tasks on non-English languages (Le et al., 2020; Rodriguez-Penagos et al., 2021; Shavrina et al., 2020) or on multiple languages (Liang et al., 2020; Hu et al., 2020). Here, we aim to improve the diversity of the lan- guages represented in NLU benchmarks by propos- ing bgGLUE, a benchmark for Bulgarian that con- sists of nine NLU tasks, including token classifica- tion, regression, and classification. Thus far, only individual datasets in Bulgarian have been used for model development and evaluation. Small subsets of up to three downstream tasks in Bulgarian have also been included in some multilingual bench- marks (Liang et al., 2020; Hu et al., 2020). Addi- tionally, there are existing benchmarks focusing on other Balto-Slavic languages, such as the Russian SuperGLUE (Shavrina et al., 2020) and the Slovene SuperGLUE (Žagar …

Content truncated to 500 words