IMPARA: Impact-Based Metric for GEC Using Parallel Data

Proceedings of the 29th International Conference on Computational Linguistics, pages 3578–3588 October 12–17, 2022. 3578 IMPARA: Impact-Based Metric for GEC Using Parallel Data Koki Maeda and Masahiro Kaneko and Naoaki Okazaki Tokyo Institute of Technology {koki.maeda@nlp., masahiro.kaneko@nlp, okazaki@}.c.titech.ac.jp Abstract Automatic evaluation of grammatical error cor- rection (GEC) is essential in developing use- ful GEC systems. Existing methods for au- tomatic evaluation require multiple reference sentences or manual scores. However, such resources are expensive, thereby hindering au- tomatic evaluation for various domains and cor- rection styles. This paper proposes an Impact- based Metric for GEC using PARAllel data, IMPARA, which utilizes correction impacts computed by parallel data comprising pairs of grammatical/ungrammatical sentences. As par- allel data is cheaper than manually assessing evaluation scores, IMPARA can reduce the cost of data creation for automatic evaluation. Cor- relations between IMPARA and human scores indicate that IMPARA is comparable or better than existing evaluation methods. Furthermore, we find that IMPARA can perform evaluations that fit different domains and correction styles trained on various parallel data. 1 Introduction Grammatical error correction (GEC) is a task of correcting grammatically incorrect sentences (Yuan and Briscoe, 2016; Chollampatt and Ng, 2018a; Junczys-Dowmunt et al., 2018; Kaneko et al., 2020, 2022; Omelianchuk et al., 2020). A GEC system is designed to be applicable in various domains, such as web text (Flachs et al., 2020) and essays written by language learners (Yannakoudakis et al., 2011) and in different correction styles, such as minimal and fluency edits (Ng et al., 2013; Napoles et al., 2017; Hotate et al., 2019). Idealy, GEC models should be evaluated by manually assessing the quality of corrections made by these models for certain target domains and styles. However, it is expensive to perform manual evaluation every time a GEC model outputs a correction; we thus need to establish an automatic evaluation method that correlates well with manual assessments. Method Fine-tuning Domain data dependence Based on language models None No With manual assessments P and M on P and M This work (IMPARA) P on P Table 1: Comparison of reference-less methods (P: par- allel data; M: manual assessment data). As discussed in Section 3, the method that is based only on language models underperforms the others. IMPARA achieves a comparable or better performance than the method trained on manual assessments, although IMPARA does not depend on manual assessment data. Automatic evaluation methods of GEC are cat- egorized into two. The first category is reference- based methods (Dahlmeier and Ng, 2012; Napoles et al., 2015; Bryant et al., 2017) that evaluate the closeness of output sentences from a GEC system and human-written reference sentences. In general, an ungrammatical sentence can be corrected in dif- ferent ways. Thus, reference-based methods re- quire multiple reference sentences for the accurate evaluation. However, Choshen and Abend (2018b) argued that it is unrealistic to prepare reference sentences that cover all possible corrections. In addition, they showed that low-coverage reference sets deteriorate the reliability of evaluation. The second category is reference-less method, which …

Content truncated to 500 words