Grammatical Error Correction as GAN-like Sequence Labeling

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3284–3290 August 1–6, 2021. ©2021 Association for Computational Linguistics 3284 Grammatical Error Correction as GAN-like Sequence Labeling Kevin Parnow1,2,3,†, Zuchao Li1,2,3,†, and Hai Zhao1,2,3∗ 1Department of Computer Science and Engineering, Shanghai Jiao Tong University 2Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China 3MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University {parnow, charlee}@sjtu.edu.cn, zhaohai@cs.sjtu.edu.cn Abstract In Grammatical Error Correction (GEC), se- quence labeling models enjoy fast inference compared to sequence-to-sequence models; however, inference in sequence labeling GEC models is an iterative process, as sentences are passed to the model for multiple rounds of cor- rection, which exposes the model to sentences with progressively fewer errors at each round. Traditional GEC models learn from sentences with fixed error rates. Coupling this with the iterative correction process causes a mismatch between training and inference that affects fi- nal performance. In order to address this mis- match, we propose a GAN-like sequence label- ing model, which consists of a grammatical er- ror detector as a discriminator and a grammat- ical error labeler with Gumbel-Softmax sam- pling as a generator. By sampling from real er- ror distributions, our errors are more genuine compared to traditional synthesized GEC er- rors, thus alleviating the aforementioned mis- match and allowing for better training. Our re- sults on several evaluation benchmarks demon- strate that our proposed approach is effec- tive and improves the previous state-of-the-art baseline. 1 Introduction Sequence-to-sequence neural solutions (Parnow et al., 2020) have been quite successful in compari- son to their statistical counterparts (Sutskever et al., 2014), but these approaches suffer from a couple key problems, which has given rise to sequence labeling approaches for GEC (Omelianchuk et al., ∗Corresponding author. † These authors made equal con- tribution. This work was supported by Huawei Noah’s Ark Lab and funded by the National Key Research and Develop- ment Program of China (No. 2017YFB0304100), the Key Projects of National Natural Science Foundation of China (U1836222 and 61733011), the Huawei-SJTU long term AI project, Cutting-edge machine reading comprehension and language model. 2020). Such approaches task models with gener- ating a list of labels to classify the grammatical errors in a sentence before correcting these errors. Sequence labeling approaches have recently gained popularity in GEC and are currently state- of-the-art. One typical aspect of sequence labeling approaches is labeling and correcting sentences through an iterative process. As successive edits will depend on how other errors are corrected in a sentence, using an iterative process and correcting only the most salient errors in each round allows models to achieve better performance; however, because of this process, models are tasked with handling sentences with varying rates of errors, as during each round of inference for a given sentence, a model encounters a sentence with progressively fewer errors. This of course causes an exposure bias problem, as the training data does not match the test data, and suggests that …

Content truncated to 500 words