HANSpeller: A Unified Framework for Chinese Spelling Correction

Computational Linguistics and Chinese Language Processing Vol. 20, No. 1, June 2015, pp. 1-22 1  The Association for Computational Linguistics and Chinese Language Processing HANSpeller: A Unified Framework for Chinese Spelling Correction Jinhua Xiong, Qiao Zhang,, Shuiyuan Zhang,, Jianpeng Hou, and Xueqi Cheng Abstract The number of people learning Chinese as a Foreign Language (CFL) has been booming in recent decades. The problem of spelling error correction for CFL learners increasingly is becoming important. Compared to the regular text spelling check task, more error types need to be considered in CFL cases. In this paper, we propose a unified framework for Chinese spelling correction. Instead of conventional methods, which focus on rules or statistics separately, our approach is based on extended HMM and ranker-based models, together with a rule-based model for further polishing, and a final decision-making step is adopted to decide whether to output the corrections or not. Experimental results on the test data of foreigner's Chinese essays provided by the SIGHAN 2014 bake-off illustrate the performance of our approach. Keywords: Chinese Spelling Correction, HMM, Ranker-Base Model, Rule-based Model, Decision-making. 1. Introduction Recent studies have shown that Chinese has become a popular choice for a second language among international college students. More and more people are learning Chinese as a Foreign Language (CFL). It is very difficult, however, for CFL learners to master Chinese because of the intrinsic linguistic features of the Chinese language. When CFL learners write Chinese essays, they are prone to generating a greater number and more diversified spelling errors than native language learners. Therefore, spelling correction tools to support such learners in  Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China E-mail: xjh@ict.ac.cn, {zhangqiao, zhangshuiyuan}@software.ict.ac.cn  University of Chinese Academy of Sciences, Beijing, China 2 Jinhua Xiong et al. correcting and polishing their Chinese essays is valuable and necessary. For the English language, there are many editing tools that provide spelling check functionality, e.g. Microsoft Word’s spellchecker. For the Chinese language, however, such tools cannot be found until now. Spelling correction has been studied for many years on regular text and web search queries. Although these two tasks share many common techniques, they have different concerns. Compared to techniques of web search query spelling correction, where corrections should be presented to search engine users in real-time, more complicated techniques can be applied to spelling correction on regular text to improve the performance, as such a situation has a lower real-time requirement. In spelling correction of Chinese essays of CFL learners, we face more challenges because of the uniqueness of the Chinese language. 1) Chinese corpora for spelling correction, especially publicly available ones, are rare, compared with English corpora. This impedes work on this practical topic. 2) There are no natural delimiters, such as spaces, between Chinese words, which may result in errors in words splitting, which may cause more splitting errors. 3) The number of error types is more than that of other cases, because CFL learners are prone to different kinds …

Content truncated to 500 words