Reassessing the Goals of Grammatical Error Correction: Fluency Instead of Grammaticality

Reassessing the Goals of Grammatical Error Correction: Fluency Instead of Grammaticality Keisuke Sakaguchi1, Courtney Napoles1, Matt Post2, and Joel Tetreault3 1Center for Language and Speech Processing, Johns Hopkins University 2Human Language Technology Center of Excellence, Johns Hopkins University 3Yahoo {keisuke,napoles,post}@cs.jhu.edu, tetreaul@yahoo-inc.com Abstract The field of grammatical error correction (GEC) has grown substantially in recent years, with research directed at both evaluation met- rics and improved system performance against those metrics. One unvisited assumption, however, is the reliance of GEC evaluation on error-coded corpora, which contain spe- cific labeled corrections. We examine cur- rent practices and show that GEC’s reliance on such corpora unnaturally constrains annota- tion and automatic evaluation, resulting in (a) sentences that do not sound acceptable to na- tive speakers and (b) system rankings that do not correlate with human judgments. In light of this, we propose an alternate approach that jettisons costly error coding in favor of unan- notated, whole-sentence rewrites. We com- pare the performance of existing metrics over different gold-standard annotations, and show that automatic evaluation with our new anno- tation scheme has very strong correlation with expert rankings (ρ = 0.82). As a result, we ad- vocate for a fundamental and necessary shift in the goal of GEC, from correcting small, la- beled error types, to producing text that has native fluency. 1 Introduction What is the purpose of grammatical error correction (GEC)? One response is that GEC aims to help peo- ple become better writers by correcting grammatical mistakes in their writing. In the NLP community, the original scope of GEC was correcting targeted error types with the goal of providing feedback to non-native writers (Chodorow and Leacock, 2000; Dale and Kilgarriff, 2011; Leacock et al., 2014). As systems improved and more advanced methods were applied to the task, the definition evolved to whole- sentence correction, or correcting all errors of every error type (Ng et al., 2014). With this pivot, we urge the community to revisit the original question. It is often the case that writing exhibits problems that are difficult to ascribe to specific grammatical categories. Consider the following example: Original: From this scope , social media has shorten our distance . Corrected: From this scope , social media has shortened our distance . If the goal is to correct verb errors, the grammat- ical mistake in the original sentence has been ad- dressed and we can move on. However, when we aim to correct the sentence as a whole, a more vex- ing problem remains. The more prominent error has to do with how unnaturally this sentence reads. The meanings of words and phrases like scope and the corrected shortened our distance are clear, but this is not how a native English speaker would use them. A more fluent version of this sentence would be the following: Fluent: From this perspective , social media has shortened the distance between us . This issue argues for a broader definition of gram- maticality that we will term native-language flu- ency, or simply …

Content truncated to 500 words