Correcting Chinese Spelling Errors with Phonetic Pre-training

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2250–2261 August 1–6, 2021. ©2021 Association for Computational Linguistics 2250 Correcting Chinese Spelling Errors with Phonetic Pre-training Ruiqing Zhang, Chao Pang, Chuanqiang Zhang, Shuohuan Wang, Zhongjun He∗, Yu Sun, Hua Wu and Haifeng Wang Baidu Inc. No. 10, Shangdi 10th Street, Beijing, 100085, China {zhangruiqing01, pangchao04, zhangchuanqiang, wangshuohuan}@baidu.com {hezhongjun, sunyu02, wu hua, wanghaifeng}@baidu.com Abstract Chinese spelling correction (CSC) is an im- portant yet challenging task. Existing state-of- the-art methods either only use a pre-trained language model or incorporate phonological information as external knowledge. In this paper, we propose a novel end-to-end CSC model that integrates phonetic features into language model by leveraging the powerful pre-training and fine-tuning method. Instead of conventionally masking words with a spe- cial token in training language model, we re- place words with phonetic features and their sound-alike words. We further propose an adaptive weighted objective to jointly train er- ror detection and correction in a unified frame- work. Experimental results show that our model achieves significant improvements on SIGHAN datasets and outperforms the previ- ous state-of-the-art methods. 1 Introduction Spelling errors are common in practice and the errors will be enlarged in the downstream tasks. Therefore, Spelling correction is important to many NLP applications such as search optimization (Mar- tins and Silva, 2004; Gao et al., 2010), machine translation (Belinkov and Bisk, 2017), part-of- speech tagging (Van Rooy and Sch¨afer, 2002; Sak- aguchi et al., 2012), etc. Spelling correction re- quires a comprehensive grasp of word similarity, language modeling and reasoning, making it one of the most challenging tasks in NLP. In this paper, we focus on Chinese spelling correction (CSC). Unlike alphabetic languages, Chinese characters cannot be typed without the help of input systems, such as Chinese Pinyin (a pronunciation-based input method) or automatic speech recognition (ASR). Thus typos of similarly pronounced characters occur quite often in Chinese text. According to Liu et al. (2010), 83% of Chi- nese spelling errors on the Internet results from ∗ Corresponding author. Wrong: ta de yu shuo de hen hao � � � � � � � He of language speak very good Correct: ta de yu shuo de hen hao � � � � � � � He German language speak very good Predict without phonetic features: ta ying yu shuo de hen hao � � � � � � � He English language speak very good Figure 1: An example of CSC. Each row contains three lines, Chinese pinyin, Chinese character, and gloss. The Chinese character “德(de, German) is incorrectly typed as its homophone “的(de, of)”. The CSC model produces a fluent but incorrect sentence by replacing the character with “英(ying, English)”, without consid- ering the phonetic similarity. phonologically similar characters. As illustrated in Figure 1, the character “德(de, German)” is in- correctly typed as one of its homophone1 “的(de, of)”. Traditional methods of CSC firstly detect mis- spelled characters and generate candidates via a language model, and then use a phonetic model or rules …

Content truncated to 500 words