How Far are We from Fully Automatic High Quality Grammatical Error Correction?

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 697–707, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics How Far are We from Fully Automatic High Quality Grammatical Error Correction? Christopher Bryant Department of Computer Science National University of Singapore 13 Computing Drive Singapore 117417 bryant@comp.nus.edu.sg Hwee Tou Ng Department of Computer Science National University of Singapore 13 Computing Drive Singapore 117417 nght@comp.nus.edu.sg Abstract In this paper, we first explore the role of inter-annotator agreement statistics in grammatical error correction and conclude that they are less informative in fields where there may be more than one correct answer. We next created a dataset of 50 student essays, each corrected by 10 dif- ferent annotators for all error types, and in- vestigated how both human and GEC sys- tem scores vary when different combina- tions of these annotations are used as the gold standard. Upon learning that even hu- mans are unable to score higher than 75% F0.5, we propose a new metric based on the ratio between human and system per- formance. We also use this method to in- vestigate the extent to which annotators agree on certain error categories, and find that similar results can be obtained from a smaller subset of just 10 essays. 1 Introduction Interest in grammatical error correction (GEC) systems has grown considerably in the past few years, thanks mainly to the success of the recent Helping Our Own (HOO) (Dale and Kilgarriff, 2011; Dale et al., 2012) and Conference on Natu- ral Language Learning (CoNLL) (Ng et al., 2013; Ng et al., 2014) shared tasks. Despite this increas- ing attention, however, one of the most significant challenges facing GEC today is the lack of a robust evaluation practice. In fact Chodorow et al. (2012) even go as far to say that it is sometimes “hard to draw meaningful comparisons between differ- ent approaches, even when they are evaluated on the same corpus.” One of the reasons for this is that, tradition- ally, system performance has only ever been eval- uated against the gold standard annotations of a single native speaker (rarely, two native speakers). As such, system output is not actually scored on the basis of grammatical acceptability alone, but rather is also constrained by the idiosyncrasies of the particular annotators. The obvious solution to this problem would be to compare systems against the gold standard an- notations of multiple annotators, in an effort to di- lute the effect of individual annotator bias, how- ever creating manual annotations is often consid- ered too time consuming and expensive. In spite of this, while other studies have instead elected to use crowdsourcing to produce multiply-corrected an- notations, often concerning only a limited number of error types (Madnani et al., 2011; Pavlick et al., 2014; Tetreault et al., 2014), one of the main con- tributions of this paper is the provision of a dataset of 10 human expert annotations, annotated in …

Content truncated to 500 words