Mitigating Exposure Bias in Grammatical Error Correction with Data Augmentation and Reweighting

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2123–2135 May 2-6, 2023 ©2023 Association for Computational Linguistics Mitigating Exposure Bias in Grammatical Error Correction with Data Augmentation and Reweighting Hannan Cao* Wenmian Yang† Hwee Tou Ng* * Department of Computer Science, National University of Singapore † School of Computer Science and Engineering, Nanyang Technological University caoh@u.nus.edu, wenmian.yang@ntu.edu.sg, nght@comp.nus.edu.sg Abstract The most popular approach in grammatical er- ror correction (GEC) is based on sequence-to- sequence (seq2seq) models. Similar to other autoregressive generation tasks, seq2seq GEC also faces the exposure bias problem, i.e., the context tokens are drawn from different distri- butions during training and testing, caused by the teacher forcing mechanism. In this paper, we propose a novel data manipulation approach to overcome this problem, which includes a data augmentation method during training to mimic the decoder input at inference time, and a data reweighting method to automatically bal- ance the importance of each kind of augmented samples. Experimental results on benchmark GEC datasets show that our method achieves significant improvements compared to prior ap- proaches.1 1 Introduction Grammatical error correction (GEC) is the task of correcting errors in a source sentence and gen- erating a well-written and grammatically correct target sentence. Good results have been achieved by state-of-the-art GEC systems (Rothe et al., 2021; Stahlberg and Kumar, 2021) based on the sequence- to-sequence (seq2seq) transformer (Vaswani et al., 2017) architecture. Generally, the seq2seq models are optimized to predict the next token given all previous context to- kens at each time step. During training, the ground truth tokens are used as the context, i.e., the so- called teacher forcing mechanism. However, dur- ing inference, the context is the previous tokens predicted by the model. As a result, the contexts at training and inference time are drawn from dif- ferent distributions. This discrepancy is called ex- posure bias (Ranzato et al., 2016), which causes the model to make inferences under conditions that 1The source code of this paper is publicly available at https://github.com/nusnlp/gec_eb Source If that also word for you, pleas close the ticker. Target If that also works for you, please close the ticket. Decoder input Actual If that also work for you, pleas close the ticker. Noise injection It that also works for you, pleas cloze the ticker. Random sampling If that also works for you, please close the ticker. Table 1: Example decoder inputs produced by different methods. Each method’s deviations from the actual decoder input are shown in bold. it has not met during training and causes errors to accumulate during inference. Currently, the exposure bias problem has at- tracted much attention in autoregressive text gen- eration. For example, in neural machine transla- tion (NMT), some work (Ranzato et al., 2016; Mi- haylova and Martins, 2019; Zhang et al., 2019) addresses this problem by utilizing sentence-level metrics, randomly injecting noise into the train- ing samples as the augmented decoder input, or randomly replacing tokens in the training samples with predicted …

Content truncated to 500 words