How Good (really) are Grammatical Error Correction Systems?

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2686–2698 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2686 How Good (really) are Grammatical Error Correction Systems? Alla Rozovskaya Queens College, CUNY arozovskaya@qc.cuny.edu Dan Roth University of Pennsylvania danroth@seas.upenn.edu Abstract Standard evaluations of Grammatical Error Correction (GEC) systems make use of a fixed reference text generated relative to the original text; they show, even when using multiple ref- erences, that we have a long way to go. This analysis paper studies the performance of GEC systems relative to closest-gold – a gold refer- ence text created relative to the output of a sys- tem. Surprisingly, we show that the real per- formance is 20-40 points better than standard evaluations show. Moreover, the performance remains high even when considering any of the top-10 hypotheses produced by a system. Importantly, the type of mistakes corrected by lower-ranked hypotheses differs in interesting ways from the top one, providing an oppor- tunity to focus on a range of errors – local spelling and grammar edits vs. more complex lexical improvements. Our study shows these results in English and Russian, and thus pro- vides a preliminary proposal for a more realis- tic evaluation of GEC systems. 1 Introduction Grammatical Error Correction (GEC) systems are typically evaluated using reference-based evalua- tion measures. This is common in language gen- eration tasks, where the system output is compared against a set of gold references, such as the set of correct translations in Machine Translation or the set of valid corrections for a source sentence in GEC. Importantly, the references are generated relative to the original text and are independent of the system outputs. In GEC, the space of valid out- puts for a given source sentence is very large, mak- ing it extremely difficult to evaluate. Specifically, reference-based evaluations (most GEC datasets contain one reference correction) are known to un- derestimate system performance (Chodorow et al., 2012; Felice and Briscoe, 2015; Bryant and Ng, 2015). Bryant and Ng (2015) showed that using Figure 1: Performance by hypothesis rank (F-score) against Reference Gold (RG) vs. Closest Golds (CGs) generated for each hypothesis. Observe dramatic drop in performance between top hypothesis and the rest in RG evaluation, vs. stability in CG evaluation; and, large gaps between scores even for the top hypotheses in RG vs. CG evaluations. two references is better than one, and the results improve further with more references; however, they used references that were generated relative to the original text. Choshen and Abend (2018b) further demonstrated that the issue can be only slightly alleviated but not completely solved by increasing the number of references. This is be- cause many errors have a long-tailed distribution on valid corrections. One can expect that as GEC systems mature and manage to address more com- plex errors, the underestimation of their perfor- mance will be further exacerbated. Choshen and Abend (2018b) discuss another consequence of having a large space of valid cor- …

Content truncated to 500 words