Cross-Corpora Evaluation and Analysis of Grammatical Error Correction Models — Is Single-Corpus Evaluation Enough?

Proceedings of NAACL-HLT 2019, pages 1309–1314 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1309 Cross-Corpora Evaluation and Analysis of Grammatical Error Correction Models — Is Single-Corpus Evaluation Enough? Masato Mita1,2,Tomoya Mizumoto1,Masahiro Kaneko3,1,Ryo Nagata4,1,Kentaro Inui2,1 1RIKEN AIP, 2Tohoku University, 3Tokyo Metropolitan University, 4Konan University {masato.mita, tomoya.mizumoto}@riken.jp, kaneko-masahiro@ed.tmu.ac.jp,nagata-naacl@ml.hyogo-u.ac.jp., inui@ecei.tohoku.ac.jp Abstract This study explores the necessity of perform- ing cross-corpora evaluation for grammati- cal error correction (GEC) models. GEC models have been previously evaluated based on a single commonly applied corpus: the CoNLL-2014 benchmark. However, the eval- uation remains incomplete because the task difficulty varies depending on the test cor- pus and conditions such as the proficiency levels of the writers and essay topics. To overcome this limitation, we evaluate the per- formance of several GEC models, including NMT-based (LSTM, CNN, and transformer) and an SMT-based model, against various learner corpora (CoNLL-2013, CoNLL-2014, FCE, JFLEG, ICNALE, and KJ). Evaluation results reveal that the models’ rankings consid- erably vary depending on the corpus, indicat- ing that single-corpus evaluation is insufficient for GEC models. 1 Introduction Grammatical error correction (GEC) is the task of correcting various grammatical errors in a given text, which is typically written by non-native speakers. Previous studies focused on typical er- rors such as those in the use of articles (Han et al., 2006), prepositions (Felice and Pulman, 2008), and noun numbers (Nagata et al., 2006). Machine translation approaches are being presently ap- plied for GEC (Junczys-Dowmunt et al., 2018; Chollampatt and Ng, 2018; Ge et al., 2018; Junczys-Dowmunt and Grundkiewicz, 2016). In these approaches, GEC is treated as a translation problem from the erroneous text to the correct text (Mizumoto et al., 2012; Felice et al., 2014; Junczys-Dowmunt and Grundkiewicz, 2014). However, the evaluation of GEC performance is unfortunately not complete because researchers tend to evaluate their models on a single corpus. The CoNLL-2014 shared task dataset (Ng et al., 2014) has been recently used for such evaluation. Single-corpus evaluation may be insufficient in cases wherein a GEC model generally aims to robustly correct grammatical errors in any writ- ten text partly because the task difficulty varies depending on proficiency levels and essay top- ics. Although a model outperforms a baseline in one corpus, the model in another corpus may per- form better, leading to different conclusions from what we know. This study explores the necessity of performing cross-corpora evaluation for GEC models. The performance of four recent models, namely three neural machine translation (NMT)- based models (LSTM, CNN, and transformer) and a statistical machine translation (SMT)- based model is evaluated against six learner corpora (CoNLL-2014, CoNLL-2013 (Ng et al., 2013), FCE (Yannakoudakis et al., 2011), JF- LEG (Napoles et al., 2017), KJ (Nagata et al., 2011), and ICNLAE (Ishikawa, 2013)). Evalua- tion results show that the models’ rankings con- siderably vary depending on the corpus. Empirical results reveal that models must be evaluated using multiple corpora from different perspectives. The contributions of this study are as follows: • We first explore the necessity …

Content truncated to 500 words