Human Evaluation of Grammatical Error Correction Systems

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 461–470, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Human Evaluation of Grammatical Error Correction Systems Roman Grundkiewicz1 and Marcin Junczys-Dowmunt1 and Edward Gillian2 1Information Systems Laboratory Faculty of Mathematics and Computer Science, Adam Mickiewicz University {romang,junczys}@amu.edu.pl 2Faculty of English, Adam Mickiewicz University egillian@pwsz.pl Abstract The paper presents the results of the first large-scale human evaluation of automatic grammatical error correction (GEC) sys- tems. Twelve participating systems and the unchanged input of the CoNLL-2014 shared task have been reassessed in a WMT-inspired human evaluation proce- dure. Methods introduced for the Work- shop of Machine Translation evaluation campaigns have been adapted to GEC and extended where necessary. The produced rankings are used to evaluate standard metrics for grammatical error correction in terms of correlation with human judgment. 1 Introduction The field of automatic grammatical error correc- tion (GEC) has seen a number of shared tasks of different scope and for different languages. The most impactful were the CoNLL-2013 and CoNLL-2014 (Ng et al., 2013; Ng et al., 2014) shared tasks on Grammatical Error Correction for ESL (English as a second language) learn- ers. They were preceded by the HOO shared tasks (Dale and Kilgarriff, 2011; Dale et al., 2012). Shared tasks for other languages took place as well, including the QALB workshops for Arabic (Mohit et al., 2014) and NLP-TEA competitions for Chinese. These tasks use automatic metrics to determine the quality of the participating systems. However, these efforts pale in comparison to competitions organized in other fields, e.g. dur- ing the annual Workshops for Machine Transla- tion (WMT). It is a central idea of the WMTs that automatic measures of machine translation quality are an imperfect substitute for human assessments. Therefore, manual evaluation of the system out- puts are conducted and their results are reported as the final rankings of the workshops. These human evaluation campaigns are an important driving fac- tor for the advancement of MT and produce in- sightful “by-products”, such as a huge number of human assessments of machine translation outputs that have been used to evaluate automatic metrics. We believe that the unavailability of this kind of quality assessment may stall the development of GEC, as all the shared tasks and the entire field have to cope with an inherent uncertainty of their methods and metrics. We hope to make a step to- wards alleviating this lack of confidence by pre- senting the results of the first1 large-scale human evaluation of automatic grammatical error correc- tion systems submitted to the CoNLL-2014 shared task. Most of our inspiration is drawn from the re- cent WMT edition (Bojar et al., 2014) and its met- rics task (Macháˇcek and Bojar, 2014). We also provide an analysis of correlation be- tween the standard metrics in GEC and human judgment and show that the commonly used pa- rameters for standard metrics in the shared task may not be optimal. The uncertainty about met- rics quality leads …

Content truncated to 500 words