Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task

Proceedings of NAACL-HLT 2018, pages 595–606 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task Marcin Junczys-Dowmunt Microsoft marcinjd@microsoft.com Roman Grundkiewicz University of Edinburgh rgrundki@inf.ed.ac.uk Shubha Guha University of Edinburgh sguha@ed-alumni.net Kenneth Heafield University of Edinburgh kheafiel@inf.ed.ac.uk Abstract Previously, neural methods in grammatical er- ror correction (GEC) did not reach state-of- the-art results compared to phrase-based sta- tistical machine translation (SMT) baselines. We demonstrate parallels between neural GEC and low-resource neural MT and successfully adapt several methods from low-resource MT to neural GEC. We further establish guide- lines for trustable results in neural GEC and propose a set of model-independent methods for neural GEC that can be easily applied in most GEC settings. Proposed methods include adding source-side noise, domain-adaptation techniques, a GEC-specific training-objective, transfer learning with monolingual data, and ensembling of independently trained GEC models and language models. The combined effects of these methods result in better than state-of-the-art neural GEC models that out- perform previously best neural GEC systems by more than 10% M2 on the CoNLL-2014 benchmark and 5.9% on the JFLEG test set. Non-neural state-of-the-art systems are outper- formed by more than 2% on the CoNLL-2014 benchmark and by 4% on JFLEG. 1 Introduction Most successful approaches to automated grammat- ical error correction (GEC) are based on methods from statistical machine translation (SMT), espe- cially the phrase-based variant. For the CoNLL 2014 benchmark on grammatical error correction (Ng et al., 2014), Junczys-Dowmunt and Grund- kiewicz (2016) established a set of methods for GEC by SMT that remain state-of-the-art. Systems (Chollampatt and Ng, 2017; Yannakoudakis et al., 2017) that improve on results by Junczys-Dowmunt and Grundkiewicz (2016) use their set-up as a back- bone for more complex systems. The view that GEC can be approached as a ma- chine translation problem by translating from erro- neous to correct text originates from Brockett et al. (2006) and resulted in many systems (e.g. Felice et al., 2014; Susanto et al., 2014) that represented the current state-of-the-art at the time. In the field of machine translation proper, the emergence of neural sequence-to-sequence meth- ods and their impressive results have lead to a paradigm shift away from phrase-based SMT to- wards neural machine translation (NMT). During WMT 2017 (Bojar et al., 2017) authors of pure phrase-based systems offered “unconditional sur- render”1 to NMT-based methods. Based on these developments, one would expect to see a rise of state-of-the-art neural methods for GEC, but as Junczys-Dowmunt and Grundkiewicz (2016) already noted, this is not the case. Interest- ingly, even today, the top systems on established GEC benchmarks are still mostly phrase-based or hybrid systems (Chollampatt and Ng, 2017; Yan- nakoudakis et al., 2017; Napoles and Callison- Burch, 2017). The best “pure” neural systems (Ji et al., 2017; Sakaguchi et al., 2017; Schmaltz et al., 2017) are several percent behind.2 If we look at recent MT work with this in mind, we find one area where phrased-based SMT domi- …

Content truncated to 500 words