A Study on Chinese Spelling Check Using Confusion Sets and?N-gram Statistics

Computational Linguistics and Chinese Language Processing Vol. 20, No. 1, June 2015, pp. 23-48 23  The Association for Computational Linguistics and Chinese Language Processing A Study on Chinese Spelling Check Using Confusion Sets and N-gram Statistics Chuan-Jie Lin and Wei-Cheng Chu Abstract This paper proposes an automatic method to build a Chinese spelling check system. Confusion sets were expanded by using two language resources, Shuowen Jiezi and the Four-Corner codes, which improved the coverages of the confusion sets. Nine scoring functions which utilize the frequency data in the Google Ngram Datasets were proposed, where the idea of smoothing was also adopted. Thresholds were also decided in an automatic way. The final system achieved far better than our baseline system in CSC 2013 Evaluation Task. Keywords: Chinese Spelling Check, Confusion Set Expansion, Google Ngram Scoring Function. 1. Introduction Automatic spelling check is a basic and important technique in building NLP systems. It has been studied since 1960s as Blair (1960) and Damerau (1964) made the first attempt to solve the spelling error problem in English. Spelling errors in English can be grouped into two classes: non-word spelling errors and real-word spelling errors. A non-word spelling error occurs when the written string cannot be found in a dictionary, such as in “fly fron* Paris”. The typical approach is finding a list of candidates from a large dictionary by edit distance or phonetic similarity (Mitton, 1996; Deorowicz & Ciura, 2005; Carlson & Fette, 2007; Chen et al., 2007; Mitton, 2008; Whitelaw et al., 2009). A real-word spelling error occurs when one word is mistakenly used for another word, such as in “fly form* Paris”. Typical approaches include using confusion set (Golding & Roth, 1999; Carlson et al., 2001), contextual information (Verberne, 2002; Islam & Inkpen, 2009), and others (Pirinen & Linden, 2010; Amorim & Zampieri, 2013). Spelling error problem in Chinese is quite different. Because there is no word delimiter  Department of Computer Science and Engineering, National Taiwan Ocean University No. 2, Pei-Ning Road, Keelung, 20224 Taiwan E-mail: (cjlin, wcchu.cse)@ntou.edu.tw 24 Chuan-Jie Lin & Wei-Cheng Chu in a Chinese sentence and almost every Chinese character can be considered as a one-character word, most of the errors are real-word errors. Although that an illegal-character error can happen where writing by hand, i.e. the written symbol is not a legal Chinese character and thus not collected in a dictionary, such an error cannot happen in a digital document because only legal Chinese characters can be typed or shown in computer. Spelling error problem in Chinese is defined as follows: given a sentence, find the locations of misused characters which result in wrong words, and propose the correct characters. There have been many attempts to solve the spelling error problem in Chinese (Chang, 1994; Zhang et al., 2000; Cucerzan & Brill, 2004; Li et al., 2006; Liu et al., 2008). Among them, lists of visually and phonologically similar characters play an important role in Chinese spelling check (Liu et al., 2011). Two Chinese spelling …

Content truncated to 500 words