RuCoLA: Russian Corpus of Linguistic Acceptability

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5207 - 5227 December 7-11, 2022 ©2022 Association for Computational Linguistics RuCoLA: Russian Corpus of Linguistic Acceptability Vladislav Mikhailov1∗, Tatiana Shamardina2∗, Max Ryabinin3,4∗ Alena Pestova3, Ivan Smurov2, Ekaterina Artemova5,6 1SberDevices, 2ABBYY, 3HSE University, 4Yandex, 5Huawei Noah’s Ark Lab, 6Center for Information and Language Processing (CIS), MaiNLP lab, LMU Munich, Germany Correspondence: vmikhailovhse@gmail.com Abstract Linguistic acceptability (LA) attracts the at- tention of the research community due to its many uses, such as testing the grammatical knowledge of language models and filtering implausible texts with acceptability classifiers. However, the application scope of LA in lan- guages other than English is limited due to the lack of high-quality resources. To this end, we introduce the Russian Corpus of Linguis- tic Acceptability (RuCoLA), built from the ground up under the well-established binary LA approach. RuCoLA consists of 9.8k in- domain sentences from linguistic publications and 3.6k out-of-domain sentences produced by generative models. The out-of-domain set is created to facilitate the practical use of ac- ceptability for improving language generation. Our paper describes the data collection pro- tocol and presents a fine-grained analysis of acceptability classification experiments with a range of baseline approaches. In particular, we demonstrate that the most widely used lan- guage models still fall behind humans by a large margin, especially when detecting mor- phological and semantic errors. We release RuCoLA, the code of experiments, and a pub- lic leaderboard1 to assess the linguistic com- petence of language models for Russian. 1 Introduction Recent NLP research has approached the lin- guistic competence of language models (LMs) with acceptability judgments, which reflect a sen- tence’s well-formedness and naturalness from the perspective of native speakers (Chomsky, 1965). These judgments have formed an empirical foun- dation in generative linguistics for evaluating hu- mans’ grammatical knowledge and language ac- quisition (Schütze, 1996; Sprouse, 2018). Borrowing conventions from linguistic theory, the community has put much effort into creating ∗Equal contribution. 1Available at rucola-benchmark.com Language Size % CoLA English 10.6k 70.5 ItaCoLA Italian 9.7k 85.4 RuCoLA Russian 13.4k 71.8 Table 1: Comparison of RuCoLA with related bi- nary acceptability classification benchmarks: CoLA (Warstadt et al., 2019) and ItaCoLA (Trotta et al., 2021). %=Percentage of acceptable sentences. linguistic acceptability (LA) resources to explore whether LMs acquire grammatical concepts piv- otal to human linguistic competence (Kann et al., 2019; Warstadt et al., 2019, 2020). Lately, simi- lar non-English resources have been proposed to address this question in typologically diverse lan- guages (Trotta et al., 2021; Volodina et al., 2021; Hartmann et al., 2021; Xiang et al., 2021). How- ever, the ability of LMs to perform acceptability judgments in Russian remains understudied. To this end, we introduce the Russian Corpus of Linguistic Acceptability (RuCoLA), a novel benchmark of 13.4k sentences labeled as accept- able or not. In contrast to related binary ac- ceptability classification benchmarks in Table 1, RuCoLA combines in-domain sentences manu- ally collected from linguistic literature and out- of-domain sentences produced by nine machine translation and …

Content truncated to 500 words