Grammatical Error Correction: Are We There Yet?

Proceedings of the 29th International Conference on Computational Linguistics, pages 2794–2800 October 12–17, 2022. 2794 Grammatical Error Correction: Are We There Yet? Muhammad Reza Qorib and Hwee Tou Ng Department of Computer Science, National University of Singapore {mrqorib, nght}@comp.nus.edu.sg Abstract There has been much recent progress in natu- ral language processing, and grammatical error correction (GEC) is no exception. We found that state-of-the-art GEC systems (T5 and GEC- ToR) outperform humans by a wide margin on the CoNLL-2014 test set, a benchmark GEC test corpus, as measured by the standard F0.5 evaluation metric. However, a careful exam- ination of their outputs reveals that there are still classes of errors that they fail to correct. This suggests that creating new test data that more accurately measure the true performance of GEC systems constitutes important future work. 1 Introduction Grammatical Error Correction (GEC) is a task that has many real-world applications, such as proof- reading, assisting language learners (Knutsson et al., 2003; Chollampatt et al., 2016; Nadejde and Tetreault, 2019), assisting children with develop- mental language disorder (Balthazar et al., 2020), etc. In order to measure the accuracy of a GEC system, multiple evaluation metrics have been pro- posed in the past. Ever since the CoNLL-2014 shared task (Ng et al., 2014), the F0.5 metric has been used as the standard evaluation metric for GEC. The F0.5 score has been found to have a bet- ter correlation with human judgment compared to F1 and other metrics (Grundkiewicz et al., 2015; Napoles et al., 2015; Chollampatt and Ng, 2018). As F0.5 is a reference-based metric, its compu- tation relies on comparing a system’s output sen- tences to human’s corrected sentences (called ref- erences henceforth). Typically, there are multiple ways to correct an input sentence. For example, the CoNLL-2014 (Ng et al., 2014) official test set contains 2 references for each input sentence, and the BEA-2019 (Bryant et al., 2019) test set con- tains 5 references. Choshen and Abend (2018) even predict that a short sentence may have more than 1,000 valid corrections. Thus, this evaluation scheme tends to underestimate the performance of a GEC system. Rozovskaya and Roth (2021) have reported that GEC systems can obtain higher scores if evaluated using references that are closer to a system’s outputs. The limited references mean that humans may also not reach 100% F0.5 performance, because their corrections need not be the same as the ref- erences. Bryant and Ng (2015) was the first to attempt to measure human performance on the CoNLL-2014 test set by adding 8 new references for each sentence, so that each sentence has 10 ref- erences. Then, they calculate each annotator’s per- formance by comparing his corrections to the other 9 annotators. The average F0.5 score of 72.58% from all annotators is then considered as the es- timated human-level performance. To compare a GEC system’s performance against human perfor- mance, a similar procedure is applied by taking the average of the system’s performance on the 10 sets of 9-annotator references. Recently, …

Content truncated to 500 words