New Dataset and Strong Baselines for the Grammatical Error Correction of Russian

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4103–4111 August 1–6, 2021. ©2021 Association for Computational Linguistics 4103 New Dataset and Strong Baselines for the Grammatical Error Correction of Russian Viet Anh Trinh Department of Computer Science CUNY Graduate Center New York, NY vtrinh@gradcenter.cuny.edu Alla Rozovskaya Department of Computer Science Queens College (CUNY) New York, NY arozovskaya@qc.cuny.edu Abstract Motivated by recent advancements in gram- matical error correction in English and exist- ing issues in the field, we describe a new re- source, an annotated learner corpus of Rus- sian, extracted from the Lang-8 language learn- ing website. This new dataset is benchmarked against two grammatical error correction mod- els that use state-of-the-art neural architec- tures. Results are provided on the newly- created corpus and are compared against per- formance on another, existing resource. We also evaluate the contribution of the Lang-8 training data to the grammatical error correc- tion of Russian and perform type-based analy- sis of the models. The expert annotations are available for research purposes. 1 Introduction The task of Grammatical Error Correction (GEC) is concerned with correcting various grammatical and usage errors in text. Recently, much progress has been made, especially in English GEC, within the framework of neural machine translation (NMT) approaches. In spite of the progress, the issue of building robust models in GEC has been empha- sized: Mita et al. (2019) showed that the perfor- mance of the models varies significantly across corpora and that single-corpus evaluation may be unreliable. While in English more efforts are be- ing made in this direction, in other languages, due to lack of benchmark corpora and other resources, very little work has been done. More importantly, as learner data in other language is very hard to come by and expensive to annotate, few benchmark corpora exist in other languages. We develop a benchmark corpus for Russian learner data, by providing expert quality annota- tions for a subset of the Russian subcorpus of Lang- 8, henceforth RU-Lang8 dataset. Lang-8 (Mizu- moto et al., 2012) is a dataset collected from a language learning website and partially corrected by native language volunteers. In Russian GEC, Rozovskaya and Roth (2019) recently released an annotated learner corpus, RULEC. The expert an- notations that we provide will allow researchers to use the created corpus as another evaluation bench- mark corpus for non-English GEC. As we show, RU-Lang8 is more diverse than RULEC in terms of the first language backgrounds, and the genre of writing. We benchmark two state-of-the-art neural machine translation models on the new corpus: a convolutional neural network (CNN) and a Trans- former model. The paper makes the following contributions: (1) We generate gold annotations for Lang-8 data to create an additional evaluation dataset for Rus- sian GEC, which is more diverse linguistically and contains data of different genre of writing, com- pared to the existing resource RULEC. We make the resource available for research purposes;1 (2) We provide benchmark results on this new corpus, using state-of-the-art models that are …

Content truncated to 500 words