PLOME: Pre-training with Misspelled Knowledge for Chinese Spelling Correction

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2991–3000 August 1–6, 2021. ©2021 Association for Computational Linguistics 2991 PLOME: Pre-training with Misspelled Knowledge for Chinese Spelling Correction Shulin Liu, Tao Yang, Tianchi Yue, Feng Zhang, Di Wang Tencent AI Platform Department, China {forestliu, rigorosyang, tianchiyue}@tencent.com {jayzhang, diwang}@tencent.com Abstract Chinese spelling correction (CSC) is a task to detect and correct spelling errors in texts. CSC is essentially a linguistic problem, thus the ability of language understanding is cru- cial to this task. In this paper, we propose a Pre-trained masked Language mOdel with Misspelled knowledgE (PLOME) for CSC, which jointly learns how to understand lan- guage and correct spelling errors. To this end, PLOME masks the chosen tokens with sim- ilar characters according to a confusion set rather than the fixed token “[MASK]” as in BERT. Besides character prediction, PLOME also introduces pronunciation prediction to learn the misspelled knowledge on phonic level. Moreover, phonological and visual sim- ilarity knowledge is important to this task. PLOME utilizes GRU networks to model such knowledge based on characters’ phonics and strokes. Experiments are conducted on widely used benchmarks. Our method achieves su- perior performance against state-of-the-art ap- proaches by a remarkable margin. We release the source code and pre-trained model for fur- ther use by the community1. 1 Introduction Chinese spelling correction (CSC) aims to detect and correct spelling errors in texts (Yu and Li, 2014). It is a challenging yet important task in natural language processing, which plays an im- portant role in various NLP applications such as search engine (Martins and Silva, 2004) and opti- cal character recognition (Afli et al., 2016). In Chi- nese, spelling errors can be mainly divided into two types: phonological errors and visual errors, which are separately caused by the misuse of phonologi- cally similar characters and visually similar char- acters. According to Liu et al. (2010), about 83% 1https://github.com/liushulinle/PLOME Figure 1: Examples of Chinese spelling errors. Mis- spelling characters are marked in red, and the corre- sponding phonics are given in brackets. of errors are phonological and 48% are visual. Fig- ure 1 illustrates examples of such errors. The first case is caused by the misuse of “没(gone)” and “美(beautiful)” with the same phonics, and the sec- ond case is caused by the misuse of “人(human)” and “入(enter)” with very similar shape. Chinese spelling correction is a challenging task because it requires human-level language under- standing ability to completely solve this problem (Zhang et al., 2020). Therefore, language model plays an important role in CSC. In fact, one of the mainstream solutions to this task is based on language models (Chen et al., 2013; Yu and Li, 2014; Tseng et al., 2015). Currently, the latest ap- proaches (Zhang et al., 2020; Cheng et al., 2020) are based on BERT (Devlin et al., 2019), which is a masked language model. In these approaches, (masked) language models are independently pre- trained from the …

Content truncated to 500 words