Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 793–805 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1074 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 793–805 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1074 Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction Christopher Bryant Mariano Felice Ted Briscoe ALTA Institute Computer Laboratory University of Cambridge Cambridge, UK {cjb255, mf501, ejb}@cl.cam.ac.uk Abstract Until now, error type performance for Grammatical Error Correction (GEC) sys- tems could only be measured in terms of recall because system output is not anno- tated. To overcome this problem, we in- troduce ERRANT, a grammatical ERRor ANnotation Toolkit designed to automat- ically extract edits from parallel original and corrected sentences and classify them according to a new, dataset-agnostic, rule- based framework. This not only facilitates error type evaluation at different levels of granularity, but can also be used to reduce annotator workload and standardise exist- ing GEC datasets. Human experts rated the automatic edits as “Good” or “Accept- able” in at least 95% of cases, so we ap- plied ERRANT to the system output of the CoNLL-2014 shared task to carry out a de- tailed error type analysis for the first time. 1 Introduction Grammatical Error Correction (GEC) systems are often only evaluated in terms of overall perfor- mance because system hypotheses are not anno- tated. This can be misleading however, and a sys- tem that performs poorly overall may in fact out- perform others at specific error types. This is sig- nificant because a robust specialised system is ac- tually more desirable than a mediocre general sys- tem. Without an error type analysis however, this information is completely unknown. The main aim of this paper is hence to rec- tify this situation and provide a method by which parallel error correction data can be automatically annotated with error type information. This not only facilitates error type evaluation, but can also be used to provide detailed error type feedback to non-native learners. Given that different cor- pora are also annotated according to different stan- dards, we also attempted to standardise existing datasets under a common error type framework. Our approach consists of two main steps. First, we automatically extract the edits between paral- lel original and corrected sentences by means of a linguistically-enhanced alignment algorithm (Fe- lice et al., 2016) and second, we classify them ac- cording to a new, rule-based framework that re- lies solely on dataset-agnostic information such as lemma and part-of-speech. We demonstrate the value of our approach, which we call the ERRor ANnotation Toolkit (ERRANT)1, by carrying out a detailed error type analysis of each system in the CoNLL-2014 shared task on grammatical er- ror correction (Ng et al., 2014). It is worth mentioning that despite an in- creased interest in GEC evaluation in recent years (Dahlmeier and Ng, 2012; Felice and Briscoe, 2015; Bryant and Ng, 2015; …

Content truncated to 500 words