Dynamic Negative Example Construction for Grammatical Error Correction using Contrastive Learning

Dynamic Negative Example Construction for Grammatical Error Correction using Contrastive Learning Junyi He, Junbin Zhuang, and Xia Li* Guangzhou Key Laboratory of Multilingual Intelligent Processing, School of Information Science and Technology, Guangdong University of Foreign Studies, Guangzhou, China {junyihe,junbinzhuang,xiali}@gdufs.edu.cn Abstract Grammatical error correction (GEC) aims at correcting texts with different types of grammatical errors into natural and correct forms. Due to the difference of error type distribution and er- ror density, current grammatical error correction systems may over-correct writings and produce a low precision. To address this issue, in this paper, we propose a dynamic negative example construction method for grammatical error correction using contrastive learning. The proposed method can construct sufficient negative examples with diverse grammatical errors, and can be dynamically used during model training. The constructed negative examples are beneficial for the GEC model to correct sentences precisely and suppress the model from over-correction. Ex- perimental results show that our proposed method enhances model precision, proving the effec- tiveness of our method. 1 Introduction Grammatical error correction (GEC) (Chollampatt and Ng, 2018; Kaneko et al., 2020; Kiyono et al., 2019) aims at correcting texts with different types of grammatical errors into natural and correct forms. It is an important research topic for both natural language processing and language education. Most of the current GEC systems are developed for correcting writings by learners of English as a second language (Brockett et al., 2006; Chen et al., 2020a; Chollampatt and Ng, 2018; Kaneko et al., 2020; Kiyono et al., 2019). However, GEC for native writings is also worth exploring, as texts written by native speakers may also contain grammatical errors that should be corrected for enhancement of writing quality. Currently, it is not feasible to train a GEC model specifically for correcting native writings because GEC data containing native writings are not sufficient. Therefore, native writings are often corrected by GEC models that are trained on GEC data consisting of writings by non-native speakers such as the Lang-8 (Mizumoto et al., 2011) and NUCLE (Dahlmeier et al., 2013) datasets. However, the error type distribution, error density and fluency are inconsistent between the writings by non-native and native speakers. Therefore, those GEC models may over correct sentences and produce a low precision of error correction (Flachs et al., 2020). In terms of this issue, contrastive learning (CL) (Chen et al., 2020b; Chen et al., 2020c; Gao et al., 2021; Liu and Liu, 2021) can be incorporated to help alleviate the over correction behaviour of the GEC models. The core idea is to take the over-corrected sentences as negative examples, and to effectively avoid or alleviate the problem of over correction by increasing the distance between the anchor sentence and the negative examples. So the focus is on how to construct effective negative examples for training the GEC models effectively. Previous studies about GEC models mainly focus on data augmentation for generating pseudo parallel training pairs as complement for the current insufficient GEC training data (Zhao et al., 2019; Takahashi et al., …

Content truncated to 500 words