Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation

Proceedings of the 28th International Conference on Computational Linguistics, pages 2085–2095 Barcelona, Spain (Online), December 8-13, 2020 2085 Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation Gotou Takumi†, Ryo Nagata†‡, Masato Mita‡♠, and Kazuhide Hanawa‡♠ †Konan University ‡RIKEN AIP ♠Tohoku University nagata-coling2020-cd @ hyogo-u.ac.jp. Abstract This paper presents performance measures for grammatical error correction which take into ac- count the difficulty of error correction. To the best of our knowledge, no conventional measure has such functionality despite the fact that some errors are easy to correct and others are not. The main purpose of this work is to provide a way of determining the difficulty of error correction and to motivate researchers in the domain to attack such difficult errors. The performance measures are based on the simple idea that the more systems successfully correct an error, the easier it is considered to be. This paper presents a set of algorithms to implement this idea. It evaluates the performance measures quantitatively and qualitatively on a wide variety of corpora and systems, revealing that they agree with our intuition of correction difficulty. A scorer and difficulty weight data based on the algorithms have been made available on the web. 1 Introduction This paper explores difficulty-weighted performance measures for grammatical error correction. The main purpose of this work is to try to increase the diversity of grammatical error correction systems by considering error correction difficulty. In other words, we would like to encourage researchers in the domain in tackling errors that are difficult to correct automatically. Despite recent progress in grammatical error correction performance, the conventional performance measures such as F0.5 and GLEU (Napoles et al., 2015) treat all errors equally. It should be emphasized that some errors are easier to correct than others. For example, errors in spelling1 are relatively easy to correct. In contrast, errors in a/the/ϕ selection and in tense are expected to be much more difficult because it requires wider contextual information or even the intention of the writer to correct them. This nature of the conventional measures discourages researchers from tackling difficult errors. Instead, it encourages them in focusing on frequent errors, which become dominant in the conventional measures. It will be good to have other measures that encourage researchers in attacking difficult errors regardless of frequency. From a technical point of view, it would be interesting to solve difficult problems. Among them, there should be errors that are important in terms of language learning assistance. Considering this background, this paper takes the first step toward developing performance measures that consider correction difficulty. Generally, one can define the difficulty of a problem in many ways. Our method adopts a simple idea inspired by academic tests (students take, for example). Simply, the more students successfully solve a problem, the easier it is considered to be. This idea can be adopted in grammatical error correction. That is, the more systems successfully correct an error, the easier it is considered to be. In other words, the difficulty …

Content truncated to 500 words