Corpora Generation for Grammatical Error Correction

Proceedings of NAACL-HLT 2019, pages 3291–3301 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3291 Corpora Generation for Grammatical Error Correction Jared Lichtarge∗, Chris Alberti∗, Shankar Kumar∗, Noam Shazeer∗, Niki Parmar∗, Simon Tong∗ Google Research {lichtarge,chrisalberti,shankarkumar,noam,nikip,simon}@google.com Abstract Grammatical Error Correction (GEC) has been recently modeled using the sequence- to-sequence framework. However, unlike se- quence transduction problems such as ma- chine translation, GEC suffers from the lack of plentiful parallel data. We describe two ap- proaches for generating large parallel datasets for GEC using publicly available Wikipedia data. The first method extracts source- target pairs from Wikipedia edit histories with minimal filtration heuristics, while the sec- ond method introduces noise into Wikipedia sentences via round-trip translation through bridge languages. Both strategies yield sim- ilar sized parallel corpora containing around 4B tokens. We employ an iterative decoding strategy that is tailored to the loosely super- vised nature of our constructed corpora. We demonstrate that neural GEC models trained using either type of corpora give similar per- formance. Fine-tuning these models on the Lang-8 corpus and ensembling allows us to surpass the state of the art on both the CoNLL- 2014 benchmark and the JFLEG task. We pro- vide systematic analysis that compares the two approaches to data generation and highlights the effectiveness of ensembling. ∗∗Equal contribution. Listing order is random. Jared conducted systematic experiments to determine useful vari- ants of the Wikipedia revisions corpus, pre-training and fine- tuning strategies, and iterative decoding. Chris implemented the ensemble and provided background knowledge and re- sources related to GEC. Shankar ran training and decoding experiments using round-trip translated data. Jared, Chris and Shankar wrote the paper. Noam identified Wikipedia revisions as a source of training data. Noam developed the heuristics for using the full Wikipedia revisions at scale and conducted initial experiments to train Transformer models for GEC. Noam and Niki provided guidance on training Trans- former models using the Tensor2Tensor toolkit. Simon pro- posed using round-trip translations as a source for training data, and corrupting them with common errors extracted from Wikipedia revisions. Simon generated such data for this pa- per. 1 Introduction Much progress in the Grammatical Error Correc- tion (GEC) task can be credited to approaching the problem as a translation task (Brockett et al., 2006) from an ungrammatical source language to a grammatical target language. This has enabled Neural Machine Translation (NMT) sequence-to- sequence (S2S) models and techniques to be ap- plied to the GEC task (Tao et al., 2018b; Chol- lampatt and Ng, 2018; Junczys-Dowmunt et al., 2018). However, the efficacy of NMT techniques is degraded for low-resource tasks (Koehn and Knowles, 2017). This poses difficulties for S2S approaches to GEC, as Lang-8, the largest publicly available parallel corpus, contains only ∼25M words (Mizumoto et al., 2011). Motivated by this data scarcity, we present two contrasting ap- proaches to generating parallel data for GEC that make use of publicly available English language Wikipedia revision histories12. Our first strategy is to mine real-world er- rors. We …

Content truncated to 500 words