Better Evaluation for Grammatical Error Correction

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568–572, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Better Evaluation for Grammatical Error Correction Daniel Dahlmeier1 and Hwee Tou Ng1,2 1NUS Graduate School for Integrative Sciences and Engineering 2Department of Computer Science, National University of Singapore {danielhe,nght}@comp.nus.edu.sg Abstract We present a novel method for evaluating grammatical error correction. The core of our method, which we call MaxMatch (M2), is an algorithm for efficiently computing the sequence of phrase-level edits between a source sentence and a system hypothesis that achieves the highest overlap with the gold- standard annotation. This optimal edit se- quence is subsequently scored using F1 mea- sure. We test our M2 scorer on the Helping Our Own (HOO) shared task data and show that our method results in more accurate eval- uation for grammatical error correction. 1 Introduction Progress in natural language processing (NLP) re- search is driven and measured by automatic eval- uation methods. Automatic evaluation allows fast and inexpensive feedback during development, and objective and reproducible evaluation during testing time. Grammatical error correction is an important NLP task with useful applications for second lan- guage learning. Evaluation for error correction is typically done by computing F1 measure between a set of proposed system edits and a set of human- annotated gold-standard edits (Leacock et al., 2010). Unfortunately, evaluation is complicated by the fact that the set of edit operations for a given system hypothesis is ambiguous. This is due to two reasons. First, the set of edits that transforms one string into another is not necessarily unique, even at the token level. Second, edits can consist of longer phrases which introduce additional ambiguity. To see how this can affect evaluation, consider the following source sentence and system hypothesis from the re- cent Helping Our Own (HOO) shared task (Dale and Kilgarriff, 2011) on grammatical error correction: Source : Our baseline system feeds word into PB-SMT pipeline. Hypot. : Our baseline system feeds a word into PB-SMT pipeline. The HOO evaluation script extracts the system edit (ϵ → a), i.e., inserting the article a. Unfortunately, the gold-standard annotation instead contains the ed- its (word → {a word, words}). Although the ex- tracted system edit results in the same corrected sen- tence as the first gold-standard edit option, the sys- tem hypothesis was considered to be invalid. In this work, we propose a method, called Max- Match (M2), to overcome this problem. The key idea is that if there are multiple possible ways to arrive at the same correction, the system should be eval- uated according to the set of edits that matches the gold-standard as often as possible. To this end, we propose an algorithm for efficiently computing the set of phrase-level edits with the maximum overlap with the gold standard. The edits are subsequently scored using F1 measure. We test our method in the context of the HOO shared task and show that our method results …

Content truncated to 500 words