Enabling Robust Grammatical Error Correction in New Domains: Data Sets, Metrics, and Analyses

Enabling Robust Grammatical Error Correction in New Domains: Data Sets, Metrics, and Analyses Courtney Napoles and Maria N˘adejde and Joel Tetreault Grammarly Courtney.Napoles@grammarly.com Abstract Until now, grammatical error correction (GEC) has been primarily evaluated on text written by non-native English speakers, with a focus on student essays. This paper enables GEC development on text written by native speakers by providing a new data set and metric. We present a multiple-reference test corpus for GEC that includes 4,000 sentences in two new domains (formal and informal writing by native English speakers) and 2,000 sentences from a diverse set of non-native student writing. We also collect human judgments of several GEC systems on this new test set and perform a meta-evaluation, assessing how reliable automatic metrics are across these domains. We find that commonly used GEC metrics have inconsistent performance across domains, and therefore we propose a new ensemble metric that is robust on all three domains of text. 1 Introduction Grammatical error correction (GEC) systems are evaluated with automatic metrics that compare their output to gold-standard corrections from reference corpora. Having automatic metrics that correlate well with human judgments allows rapid system development and reliable evaluations across the field. Although the GEC community has benefited from several evaluation sets over the past several years, they are primarily composed of student essays written by non-native English speakers (Yannakoudakis et al., 2011; Dahlmeier et al., 2013; Napoles et al., 2017; Bryant et al., 2019). As of yet, we do not know how well GEC systems do on other domains of text or how reliable automatic evaluation is when we move to other domains. We tackle this issue head-on by creating a new test set for GEC that represents diverse domains of text. We call this new test set Grammarly Multi-domain Evaluation for GEC Data set (GMEG-Data), and it is the first test set to include multiple corrections of sentences written by native English speakers and informal writing. We collect human ratings of the corrections generated by 6 GEC systems on this data. From this gold-standard human evaluation, we measure the performance of current GEC metrics and find that the standard metrics are not robust across domains. We propose Grammarly Multi-domain Evaluation for GEC Metric (GMEG-Metric), a new ensemble scorer that is reliable on all three domains and draws from the statistics of existing metrics. This work takes the first major step to develop truly robust GEC systems. Our contributions include: • GMEG-Data: The largest multiple-reference GEC test set with 6,000 sentences from three domains: formal and informal text by native English speakers, and a diverse set of student writing. • The output of 6 GEC systems with varying modern architectures and training data sizes, and human judgments of the entire set of system outputs. • Evaluation of 4 standard GEC metrics and a leading machine translation (MT) metric on the new test set. • GMEG-Metric: A new ensemble metric for GEC that is robust across all three domains. • GMEG-Data, GMEG-Metric, human ratings, …

Content truncated to 500 words