Construction of a Quality Estimation Dataset for Automatic Evaluation of Japanese Grammatical Error Correction

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5565–5572 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5565 Construction of a Quality Estimation Dataset for Automatic Evaluation of Japanese Grammatical Error Correction Daisuke Suzuki1, Yujin Takahashi1, Ikumi Yamashita1, Taichi Aida1, Tosho Hirasawa1, Michitaka Nakatsuji1, Masato Mita2,1, Mamoru Komachi1 Tokyo Metropolitan University1 RIKEN2 {suzuki-daisuke3@ed., takahashi-yujin@ed., yamashita-ikumi@ed., aida-taichi@ed., hirasawa-tosho@ed., nakatsuji-michitaka@ed.,}tmu.ac.jp, masato.mita@riken.jp, komachi@tmu.ac.jp Abstract In grammatical error correction (GEC), automatic evaluation of the system outputs is an important factor for the research and development of GEC systems. In this regard, existing studies on automatic evaluation have demonstrated that quality estimation models built from datasets with manual evaluation can achieve high performance in automatic evaluation of English GEC without using reference sentences. However, quality estimation models for the Japanese language have not been studied yet owing to the absence of datasets necessary for constructing such models. Therefore, in this study, we created a quality estimation dataset with manual evaluation to build an automatic evaluation model for Japanese GEC. Moreover, we conducted a meta-evaluation to verify the usefulness of this dataset in building the Japanese quality estimation model. Keywords: corpus construction, automatic evaluation, quality estimation, Japanese grammatical error correction 1. Introduction Grammatical error correction (GEC) is the task of correcting different types of errors in text such as those pertaining to spelling, punctuation, grammar, and word-choice. Automatic evaluation is considered as an important factor to enable the continuous integration and deployment of GEC systems. There are two types of automatic evaluation systems for GEC: reference-based and reference-less. Reference- based systems have been used extensively by the GEC community, whereas reference-less systems have gained considerable attention only recently. Hereafter, we refer reference-less systems as quality estimation models. One of the problems for reference-based methods is the diversity of references. In a study by (Bryant and Ng, 2015), a dataset with multiple corrections was cre- ated by 10 annotators based on the CoNLL-2014 test set (Ng et al., 2014). The authors observed that the number of error types corrected by each annotator var- ied significantly. This means that there can be a va- riety of corrected versions for a grammatically incor- rect sentence. However, with a limited number of ref- erence sentences, it is difficult to cover a large variety of corrected sentences in GEC. Therefore, reference- based automatic evaluation methods may assign unrea- sonably low scores to valid system-outputs when they are not observed amongst the references. By contrast, quality estimation models do not present this problem because they estimate the quality of the system’s output without requiring gold-standard refer- ences. Asano et al. (2017) and Yoshimura et al. (2020) built a quality estimation model using a dataset with manual evaluations of system-outputs,1 and achieved a higher correlation between manually evaluated val- ues and automatically assigned scores than that by reference-based methods. However, there is no dedicated quality estimation dataset for the Japanese language, although Japanese is popular amongst learners. As a result, no reference- less …

Content truncated to 500 words