Automatic Classification of Russian Learner Errors

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5637–5647 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5637 Automatic Classification of Russian Learner Errors Alla Rozovskaya Queens College City University of New York arozovskaya@qc.cuny.edu Abstract Grammatical Error Correction systems are typically evaluated overall, without taking into consideration performance on individual error types because system output is not annotated with respect to error type. We introduce a tool that automatically classifies errors in Russian learner texts. The tool takes an edit pair consisting of the original token(s) and the corresponding replacement and provides a grammatical error category. Manual evaluation of the output reveals that in more than 93% of cases the error categories are judged as correct or acceptable. We apply the tool to carry out a fine-grained evaluation on the performance of two error correction systems for Russian. Keywords: grammatical error correction, Russian, error classification 1. Introduction In the field of Grammatical Error Correction (GEC), similarly to Machine Translation, evaluation is per- formed by first producing edits (token deletions, in- sertions, and replacements) needed to transform the source sentence into its corrected counterpart. This is done by aligning the two sentences. A gold edit is an edit between a source sentence and its reference, i.e. a corrected version produced by a human annotator. A proposed (system) edit is an edit between a source sen- tence and the corrected version proposed by the auto- matic system, or system hypothesis. A correct edit is an edit in the intersection of gold and proposed edits. Below we show a sample (source, reference) pair, a possible alignment, and the resulting gold edits: “re- allistic” → “realistic”, “a” → ∅), and “are” → “were” . • Source: The settings are very reallistic and the actors had a great performance . • Ref.: The settings were very realistic and the ac- tors had great performance . When annotating and correcting learner data, annota- tors can also be asked to categorize the resulting gold edits. For the example above, the edits would be clas- sified as verb tense, spelling, and missing determiner. Classifying system edits generated by neural machine translation frameworks that are being used today is not trivial, as these systems are not restricted in the type of edits that can be made (Susanto et al., 2014; Yuan and Briscoe, 2016; Hoang et al., 2016; ?; Junczys- Dowmunt and Grundkiewicz, 2016; Mizumoto and Matsumoto, 2016; Jianshu et al., 2017; Chollampatt and Ng, 2018). As the system edits are not labeled for linguistic type, it is not possible to perform type-based evaluation. Type- based evaluation would be extremely useful, as it could help provide directions for making further progress in system development by allowing one to focus on and improve performance on specific error types. Further- more, the resulting error categories can be used in language learning applications, to provide educational feedback. Finally, automatic type classification allows for a standartization of multiple GEC datasets (Bryant et al., …

Content truncated to 500 words