Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8467–8478, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8467 Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses Simon Flachs1,2, Oph´elie Lacroix3∗, Helen Yannakoudakis4, Marek Rei5, Anders Søgaard2 1 Siteimprove sfl@siteimprove.com 2 University of Copenhagen soegaard@di.ku.dk 3 Alexandra Institute ophelie.lacroix@alexandra.dk 4 King’s College London helen.yannakoudakis@kcl.ac.uk 5 Imperial College London marek.rei@imperial.ac.uk Abstract Evaluation of grammatical error correction (GEC) systems has primarily focused on es- says written by non-native learners of English, which however is only part of the full spectrum of GEC applications. We aim to broaden the target domain of GEC and release CWEB, a new benchmark for GEC consisting of website text generated by English speakers of varying levels of proficiency. Website data is a com- mon and important domain that contains far fewer grammatical errors than learner essays, which we show presents a challenge to state- of-the-art GEC systems. We demonstrate that a factor behind this is the inability of systems to rely on a strong internal language model in low error density domains. We hope this work shall facilitate the development of open- domain GEC models that generalize to differ- ent topics and genres. 1 Introduction Grammatical error correction (GEC) is the task of automatically editing text to remove grammatical errors; for example: [A link to registration can also be found at on the same page.]. GEC systems so far have primarily focused on correcting essays produced by English-as-a-second-language (ESL) learners, providing fast and inexpensive feedback to facilitate language learning. However, this is only one target domain in the full spectrum of GEC applications. GEC models can also help to improve written communication outside of the formal edu- cation setting. Today the largest medium of written communication is the internet, with approximately 380 new websites created every minute.1 Ensuring grammatical correctness of websites helps facilitate clear communication and a professional commer- cial presentation. Therefore, it is important that ∗Research conducted at Siteimprove. 1https://www.millforbusiness.com/ how-many-websites-are-there/ JFLEG FCE CoNLL14 GMEG Wiki GMEG Yahoo W&I-A W&I-B W&I-C LOCNESS AESW CWEB-G CWEB-S 0 5 10 15 20 % erroneous tokens Figure 1: Percentage of erroneous tokens per domain. CWEB-G/S are our newly devised datasets. GEC models perform well in the open-domain set- ting and generalize, not only to writing produced in the educational context, but also to language production “in the wild”. Website data specifically represent a broad and diverse range of writing and constitute a major part of what people read and write on an everyday basis. This work highlights two major prevailing chal- lenges of current approaches to GEC: domain adap- tation and low precision in texts with low error den- sity. Previous work has primarily targeted essay- style text with high error density (see Figure 1); however, this lack of diversity means that it is not clear how systems perform on other domains and under different error distributions (Sakaguchi et al., 2017).2 Current publicly available datasets are restricted to non-native …

Content truncated to 500 words