Slovene SuperGLUE Benchmark: Translation and Evaluation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2058–2065 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2058 Slovene SuperGLUE Benchmark: Translation and Evaluation Aleˇs ˇZagar, Marko Robnik-ˇSikonja University of Ljubljana, Faculty of Computer and Information Science, Veˇcna pot 113, 1000 Ljubljana, ales.zagar@fri.uni-lj.si, marko.robnik@fri.uni-lj.si Abstract We present SuperGLUE benchmark adapted and translated into Slovene using a combination of human and machine translation. We describe the translation process and problems arising due to differences in morphology and grammar. We evaluate the translated datasets in several modes: monolingual, cross-lingual, and multilingual, taking into account differences between machine and human translated training sets. The results show that the monolingual Slovene SloBERTa model is superior to massively multilingual and trilingual BERT models, but these also show a good cross-lingual performance on certain tasks. The performance of Slovene models still lags behind the best English models. Keywords: benchmarks, evaluation, transformer models, natural language understanding, cross-lingual models, multi- lingual models 1. Introduction To measure the progress across the fast evolving area of natural language processing (NLP), several promi- nent benchmarking suites have been proposed, such as SentEval (Conneau et al., 2018), GLUE (Wang et al., 2019b), and SuperGLUE (Wang et al., 2019a). Sen- tEval aims to evaluate sentence encoders, while GLUE and its more challenging successor, SuperGLUE, eval- uate general language understanding. These two benchmark suites contain public training sets and pri- vate test sets that can be assessed through an eval- uation server giving separate scores for each of the tasks and an overall single-number score. The tasks in SuperGLUE are diverse and comprised of ques- tion answering (QA), natural language inference (NLI), coreference resolution, and word sense disambiguation (WSD). Non-expert humans evaluated all the tasks to provide a human performance baseline. Despite recent criticism of the way how the evalua- tion scores in these benchmarks are computed (e.g., the arithmetic mean of separate metrics is used for all tasks of different complexity and different sizes of testing sets) (Shavrina and Malykh, 2021; Kasai et al., 2021), there is little doubt that the datasets con- tained in the benchmarks significantly contribute to the progress in the NLP area. Unfortunately, most of the research is focused on English, which often lim- its the generality of the approaches and does not ad- dress the full scale of language complexity. Still, we can observe significant efforts to make relevant datasets cross-lingual or adapted to more languages. Broad multilingual benchmarking suites, such as XTREME (Hu et al., 2020) and XGLUE (Liang et al., 2020), provide machine-translated datasets for several rele- vant tasks and languages. Complementary to that, we present a combined machine-human translation of SuperGLUE benchmarking suite for less-resourced Slovene language. We describe the difficulties aris- ing in the translation and provide the first evaluation of the datasets using large pretrained BERT-like lan- guage models. Our evaluation encompasses monolin- gual, cross-lingual and multilingual approaches, com- parison of human and machine-translated parts, and comparison of two state-of-the-art machine translation systems. …

Content truncated to 500 words