ProQE: Proficiency-wise Quality Estimation dataset for Grammatical Error Correction

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5994–6000 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5994 ProQE: Proficiency-wise Quality Estimation dataset for Grammatical Error Correction Yujin Takahashi1, Masahiro Kaneko2, Masato Mita3,1, Mamoru Komachi1 1Tokyo Metropolitan University, 2Tokyo Institute of Technology, 3RIKEN takahashi-yujin@ed.tmu.ac.jp, masahiro.kaneko@nlp.c.titech.ac.jp, masato.mita@riken.jp, komachi@tmu.ac.jp Abstract This study investigates how supervised quality estimation (QE) models of grammatical error correction (GEC) are affected by the learners’ proficiency with the data. QE models for GEC evaluations in prior work have obtained a high correlation with manual evaluations. However, when functioning in a real-world context, the data used for the reported results have limitations because prior works were biased toward data by learners with relatively high proficiency levels. To address this issue, we created a QE dataset that includes multiple proficiency levels and explored the necessity of performing proficiency-wise evaluation for QE of GEC. Our experiments demonstrated that differences in evaluation dataset proficiency affect the performance of QE models, and proficiency-wise evaluation helps create more robust models. Keywords: Grammatical Error Correction, Evaluation, Quality Estimation, Proficiency, Corpus 1. Introduction Grammatical error correction (GEC) refers to the task of correcting a variety of grammatical errors in text written by non-native speakers learning a language. Thus, the main application of a GEC system is to assist language learners with their writing. The performance of GEC systems has been improving, but there is still room for improvements. The precision of the state-of- the-art system is approximately 75% and the recall is below 50% on the CoNLL-2014 test set (Stahlberg and Kumar, 2021). Therefore, it is difficult for learners to judge whether they can trust the GEC system’s output, as it may be potentially misleading in some cases. The quality estimation (QE) of GEC, which is the task of estimating the quality of GEC system outputs, has attracted attention as a key method to address the afore- mentioned difficulties in real-world scenarios (Heilman et al., 2014; Chollampatt and Ng, 2018b; Asano et al., 2017; Yoshimura et al., 2020). Quality estimates of the system’s output can help learners and instructors decide whether to adopt or ignore the system’s correc- tion. Recently, it has been reported that QE using Bidi- rectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019) was highly correlated with human evaluations (Yoshimura et al., 2020). However, the current evaluation in QE of GEC is un- fortunately insufficient because it is restricted to a cor- pus limited to certain proficiency levels. For exam- ple, the data used in prior works (Asano et al., 2017; Yoshimura et al., 2020) were biased toward data gener- ated for learners with relatively high proficiency levels, such as CoNLL-2014 (Ng et al., 2014). By contrast, it is assumed that the proficiency level of language learn- ers varies from beginners to advanced learners in real- world use cases. Given the nature of the QE task, QE systems need to estimate quality with high accuracy for sentences written by learners with …

Content truncated to 500 words