SOME: Reference-less Sub-Metrics Optimized for Manual Evaluations of Grammatical Error Correction

Proceedings of the 28th International Conference on Computational Linguistics, pages 6516–6522 Barcelona, Spain (Online), December 8-13, 2020 6516 SOME: Reference-less Sub-Metrics Optimized for Manual Evaluations of Grammatical Error Correction Ryoma Yoshimura⇤, Masahiro Kaneko⇤, Tomoyuki Kajiwara† ⇤, and Mamoru Komachi‡ ⇤Tokyo Metropolitan University, Tokyo, Japan, {yoshimura-ryoma, kaneko-masahiro}@ed.tmu.ac.jp †Osaka University, Osaka, Japan, kajiwara@ids.osaka-u.ac.jp ‡Tokyo Metropolitan University, Tokyo, Japan, komachi@tmu.ac.jp Abstract We propose a reference-less metric trained on manual evaluations of system outputs for gram- matical error correction. Previous studies have shown that reference-less metrics are promising; however, existing metrics are not optimized for manual evaluation of the system output because there is no dataset of system output with manual evaluation. This study manually evaluates the output of grammatical error correction systems to optimize the metrics. Experimental results show that the proposed metric improves the correlation with manual evaluation in both system- and sentence-level meta-evaluation. Our dataset and metric will be made publicly available.1 1 Introduction Grammatical error correction (GEC) is the task of automatically correcting grammatically incorrect sen- tences, especially those written by language learners. To develop GEC systems efficiently, we construct an evaluation metric that has a high correlation with manual evaluations. Reference-based metrics such as Max Match (M2) (Dahlmeier and Ng, 2012) and GLEU (Napoles et al., 2015) are commonly used for automatic evaluation in the GEC task. However, these metrics penalize sentences whose words or phrases are not included in the reference, even if they are correct expressions because it is difficult to cover all possible references (Choshen and Abend, 2018). In contrast, reference-less metrics (Napoles et al., 2016; Asano et al., 2017) do not suffer from this limitation. Among them, Asano et al. (2017) achieved a higher correlation with manual evaluations than reference-based metrics by integrating sub-metrics from the three perspectives of (i) grammaticality, (ii) fluency, and (iii) meaning preservation. However, the correlation with the manual evaluation of system output can be further improved because they are not considered for optimizing each sub-metric. To achieve a better correlation with manual evaluation, we create a dataset to optimize each sub- metric to the manual evaluation of GEC systems. Our annotators evaluated the output of five typical GEC systems in terms of each sub-metric of Asano et al. (2017). We propose a reference-less metric consisting of sub-metrics that are optimized for manual evaluation (SOME). It combines three regression models trained on our dataset. Experimental results show that the proposed metric improves correlation with the manual evaluation in both system- and sentence-level meta-evaluation. Detailed analysis reveals that optimization for both the manual evaluation and the output of GEC systems contribute to improvement. 2 Related Work Napoles et al. (2016) pioneered the reference-less GEC metric. They presented a metric based on gram- matical error detection tools and linguistic features such as language models, and demonstrated that its performance was close to that of reference-based metrics. Asano et al. (2017) combined three sub- metrics: grammaticality, fluency, and meaning preservation, and outperformed reference-based metrics. They trained a logistic regression model on the GUG dataset2 …

Content truncated to 500 words