Revisiting Grammatical Error Correction Evaluation and Beyond

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6891 - 6902 December 7-11, 2022 ©2022 Association for Computational Linguistics Revisiting Grammatical Error Correction Evaluation and Beyond Peiyuan Gong1 Xuebo Liu2∗ Heyan Huang1 Min Zhang2 1School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China {3120201019,hhy63}@bit.edu.cn 2Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China {liuxuebo,zhangmin2021}@hit.edu.cn Abstract Pretraining-based (PT-based) automatic evalua- tion metrics (e.g., BERTScore and BARTScore) have been widely used in several sentence gen- eration tasks (e.g., machine translation and text summarization) due to their better correlation with human judgments over traditional overlap- based methods. Although PT-based methods have become the de facto standard for train- ing grammatical error correction (GEC) sys- tems, GEC evaluation still does not benefit from pretrained knowledge. This paper takes the first step towards understanding and improv- ing GEC evaluation with pretraining. We first find that arbitrarily applying PT-based metrics to GEC evaluation brings unsatisfactory corre- lation results because of the excessive attention to inessential systems outputs (e.g., unchanged parts). To alleviate the limitation, we propose a novel GEC evaluation metric to achieve the best of both worlds, namely PT-M2, which only uses PT-based metrics to score those corrected parts. Experimental results on the CoNLL14 evaluation task show that PT-M2 significantly outperforms existing methods, achieving a new state-of-the-art result of 0.949 Pearson corre- lation. Further analysis reveals that PT-M2 is robust to evaluate competitive GEC systems. Source code and scripts are freely available at https://github.com/pygongnlp/PT-M2. 1 Introduction Grammatical error correction (GEC) is the task that takes a sentence with grammatical errors as input, and outputs a corrected sentence. Due to the important role of GEC in the field of second language learning and intelligent writing, GEC has attracted wide attention from the commu- nity (Chollampatt and Ng, 2018a; Lewis et al., 2020; Omelianchuk et al., 2020). As a typical natu- ral language generation task (NLG), the common practice for GEC evaluation is to calculate the simi- ∗Corresponding author Task Training Evaluation Machine Translation • • Text Summarization • • Grammatical Error Correction • ◦ Table 1: The use of PT-based models for model training and evaluation in various NLG tasks. • means used and ◦ means unused. Unlike other tasks, GEC has not used PT-based methods for evaluation. larity between system outputs and their correspond- ing references (Dahlmeier and Ng, 2012; Napoles et al., 2015; Bryant et al., 2017). With the rapid development of pretraining (PT) (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020), several NLG tasks, such as machine transla- tion, text summarization and GEC, have been utiliz- ing PT-based models to improve the model training process (Kaneko et al., 2020; Omelianchuk et al., 2020; Tarnavskyi et al., 2022), as shown in Table 1. Furthermore, since PT-based models can learn rich syntactic and semantic knowledge from a large amount of unlabeled data, they are able to calculate the similarity between two sentences more accu- rately. Therefore, many mainstream NLG tasks try …

Content truncated to 500 words