SpellBERT: A Lightweight Pretrained Model for Chinese Spelling Check

SpellBERT: A Lightweight Pretrained Model for Chinese Spelling Check Tuo Ji, Hang Yan, Xipeng Qiu∗ School of Computer Science, Fudan University Shanghai Key Laboratory of Intelligent Information Processing, Fudan University {tji19, hyan19, xpqiu}@fudan.edu.cn Abstract Chinese Spelling Check (CSC) is to detect and correct Chinese spelling errors. Many mod- els utilize a predefined confusion set to learn a mapping between correct characters and its visually similar or phonetically similar misuses but the mapping may be out-of-domain. To that end, we propose SpellBERT, a pretrained model with graph-based extra features and inde- pendent on confusion set. To explicitly capture the two erroneous patterns, we employ a graph neural network to introduce radical and pinyin information as visual and phonetic features. For better fusing these features with character repre- sentations, we devise masked language model alike pre-training tasks. With this feature-rich pre-training, SpellBERT with only half size of BERT can show competitive performance and make a state-of-the-art result on the OCR dataset where most of the errors are not covered by the existing confusion set 1 Introduction Spelling Check is to detect and correct Chinese spelling errors in sentences. However, it is a non- trivial task for Chinese spelling check because of the nature of ideographic language. Chinese has a large vocabulary including at least 3,500 common characters which leads to huge search space and an unbalanced distribution of errors. Though hard to cover most of the misuses, their patterns could be roughly reduced to visual or pho- netic errors (Chang, 1995) as shown in Figure 1. The former type of errors have similar shapes as correct ones and they are often caused by optical character recognition (OCR) or morphology-based input method. The other type of errors have similar pronunciation as original ones and they are usually caused by automatic speech recognition (ASR) or phonetic-based input method. Previous work (Hsieh et al., 2013; Yu and Li, 2014; Wang et al., 2019a; Cheng et al., 2020) tend ∗Corresponding Author. 我喜欢吃蛋高 糕 我喜欢吃蛋达 挞 Pinyin: g ao Radical : ⿺ 辶 大 ⿰ 扌 (a) ✔️(b) ❌ ❌ ✔️ Figure 1: The two erroneous patterns. (a) is a phonetic error and its pinyin have overlap with the correct ones. (b) is a visual error and its radicals also have overlap to employ a predefined confusion set to find and filter correction candidates. Confusion set is con- structed by incorrect stats (Liu et al., 2010) and it has a mapping between visually similar pairs and phonetically similar pairs in accord with erro- neous patterns. However, these models only learn a shallow mapping from confusion set and their performance is heavily dependent on the quality of confusion set. But it is hard to find an up-to-date and in-domain confusion set. In this paper, we devise two pre-training tasks to model the two aforementioned erroneous pat- terns explicitly. To model visual errors, we intro- duce radical features. Chinese characters can be decomposed into various components namely rad- ical. As for phonetic errors, we employ pinyin as features which …

Content truncated to 500 words