RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4717–4726, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4717 RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark Tatiana Shavrina1,2, Alena Fenogenova1, Anton Emelyanov1,3, Denis Shevelev1, Ekaterina Artemova2, Valentin Malykh4, Vladislav Mikhailov1,2, Maria Tikhonova1,2, Andrey Chertok1 and Andrey Evlampiev1 1Sberbank / Moscow, Russia 2National Research University Higher School of Economics / Moscow, Russia 3Moscow Institute of Physics and Technology / Moscow, Russia 4Huawei / Moscow, Russia Abstract In this paper, we introduce an advanced Rus- sian general language understanding evalua- tion benchmark – RussianGLUE. Recent advances in the field of universal lan- guage models and transformers require the de- velopment of a methodology for their broad di- agnostics and testing for general intellectual skills - detection of natural language infer- ence, commonsense reasoning, ability to per- form simple logical operations regardless of text subject or lexicon. For the first time, a benchmark of nine tasks, collected and orga- nized analogically to the SuperGLUE method- ology (Wang et al., 2019), was developed from scratch for the Russian language. We provide baselines, human level evaluation, an open- source framework for evaluating models and an overall leaderboard of transformer models for the Russian language. Besides, we present the first results of compar- ing multilingual models in the adapted diag- nostic test set and offer the first steps to further expanding or assessing state-of-the-art models independently of language. 1 Introduction With the development of technologies for text pro- cessing and then deep learning methods for ob- taining better text representation, language models went through the increasingly advanced stages of natural language modelling. Modern scientific methodology is beginning to gradually explore universal transformers as an inde- pendent object of study - furthermore, such models show the ability to extract causal relationships in texts (natural language inference), common sense and world knowledge and logic (textual entail- ment), to generate coherent and correct texts. An actively developing field of model interpretation develops testing procedures comparing their per- formance to a human level and even the ability to reproduce some mechanisms of human brain func- tions. NLP is gradually absorbing all the new areas responsible for the mechanisms of thinking and the theory of artificial intelligence. Benchmark approaches are being developed, testing general intellectual “abilities” in a text for- mat, including complex input content, but having a simple output format. Most of these benchmarks (for more details see Section 2) make the develop- ment of machine intelligence anglo-centric, while other, less widespread languages, in particular Rus- sian, have other characteristic linguistic categories to be tested. In this paper, we expand the linguistic diversity of the testing methodology and present the first benchmark for evaluating universal language mod- els and transformers for the Russian language, to- gether with a portable methodology for collecting and filtering the data for other languages. The contribution of RussianGLUE is two-fold. First, it provides nine novel datasets for the Russian language covering a wide scope of NLU tasks. The choice of the …

Content truncated to 500 words