Self-Supervised Curriculum Learning for Spelling Error Correction

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3487–3494 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3487 Self-Supervised Curriculum Learning for Spelling Error Correction Zifa Gan1 Hongfei Xu2 Hongying Zan1∗ 1Zhengzhou University, Henan, China 2DFKI and Saarland University, Informatics Campus, Saarland, Germany {zfganlp, hfxunlp}@foxmail.com, iehyzan@zzu.edu.cn Abstract Spelling Error Correction (SEC) that requires high-level language understanding is a chal- lenging but useful task. Current SEC ap- proaches normally leverage a pre-training then fine-tuning procedure that treats data equally. By contrast, Curriculum Learning (CL) uti- lizes training data differently during training and has shown its effectiveness in improving both performance and training efficiency in many other NLP tasks. In NMT, a model’s per- formance has been shown sensitive to the dif- ficulty of training examples, and CL has been shown effective to address this. In SEC, the data from different language learners are nat- urally distributed at different difficulty levels (some errors made by beginners are obvious to correct while some made by fluent speak- ers are hard), and we expect that designing a curriculum correspondingly for model learn- ing may also help its training and bring about better performance. In this paper, we study how to further improve the performance of the state-of-the-art SEC method with CL, and pro- pose a Self-Supervised Curriculum Learning (SSCL) approach. Specifically, we directly use the cross-entropy loss as criteria for: 1) scoring the difficulty of training data, and 2) evaluating the competence of the model. In our approach, CL improves the model train- ing, which in return improves the CL measure- ment. In our experiments on the SIGHAN 2015 Chinese spelling check task, we show that SSCL is superior to previous norm-based and uncertainty-aware approaches, and estab- lish a new state of the art (74.38% F1). 1 Introduction Spelling Error Correction (SEC) aims to automat- ically correct the spelling errors in written text ei- ther at word-level or character-level (Yu and Li, 2014; Yu et al., 2014; Zhang et al., 2015; Wang et al., 2018; Hong et al., 2019; Wang et al., 2019a). ∗ Corresponding author. Although being a very valuable natural language application, SEC is a challenging task and needs high-level language understanding. Curriculum Learning (CL) (Bengio et al., 2009) facilitates model training in an easy-to-hard order. Previous studies (Kocmi and Bojar, 2017; Platan- ios et al., 2019; Zhang et al., 2019; Zhou et al., 2020) use sentence length or word rarity for CL, but merely consider features over sentences, which is not capable to fully reflect the data challenge for a model. SEC data difficulty is influenced by many factors, such as sentence length, word rarity and a great diversity of errors. In addition, previous CL approaches require careful design for data dif- ficulty and training curricula. Ruiter et al. (2020) show that self-supervised learning is a curriculum learner, which might be useful to avoid such efforts. In this paper, we propose a novel Self-Supervised CL (SSCL) approach to evaluating data difficulty from the model’s perspective and …

Content truncated to 500 words