The Properties and Further Applications of Chinese Frequent Strings

Computational Linguistics and Chinese Language Processing Vol. 9, No. 1, February 2004 , pp.113-128 113 © The Association for Computational Linguistics and Chinese Language Processing The Properties and Further Applications of Chinese Frequent Strings Yih-Jeng Lin*, and Ming-Shing Yu+ Abstract This paper reveals some important properties of CFSs and applications in Chinese natural language processing (NLP). We have previously proposed a method for extracting Chinese frequent strings that contain unknown words from a Chinese corpus [Lin and Yu 2001]. We found that CFSs contain many 4-character strings, 3-word strings, and longer n-grams. Such information can only be derived from an extremely large corpus using a traditional language model(LM). In contrast to using a traditional LM, we can achieve high precision and efficiency by using CFSs to solve Chinese toneless phoneme-to-character conversion and to correct Chinese spelling errors with a small training corpus. An accuracy rate of 92.86% was achieved for Chinese toneless phoneme-to-character conversion, and an accuracy rate of 87.32% was achieved for Chinese spelling error correction. We also attempted to assign syntactic categories to a CFS. The accuracy rate for assigning syntactic categories to the CFSs was 88.53% for outside testing when the syntactic categories of the highest level were used. Keywords: Chinese frequent strings, unknown words, Chinese toneless phoneme-to-character, Chinese spelling error correction, language model. 1. Introduction An increasing number of new or unknown words are being used on the Internet. Such new or unknown words are called “out of vocabulary (OOV) words” [Yang 1998], and they are not listed in traditional dictionaries. Many researchers have overcome problems caused by OOV words by using N-gram LMs along with smoothing methods. N-gram LMs have many useful applications in NLP [Yang 1998]. In Chinese NLP tasks, word-based bi-gram LMs are used by many researchers. To obtain useful probabilities for training, a corpus size proportional to 800002 (80000 is the approximate number of words in ASCED) = 6.4*109 words is required. * Department of Information Management, Chien Kuo Institute of Technology, Changhua, 500 Taiwan E-mail: yclin@ckit.edu.tw Tel: 04-7111111 ext 3637 Fax:04-7111142 + Department of Computer Science, National Chung-Hsing University, Taichung, 40227 Taiwan 114 Yih-Jeng Lin, and Ming-Shing Yu However, it is not easy to find such a corpus at the present time. A small-size corpus will lead too many unseen events when using N-gram LMs. Although we can apply some smoothing strategies, such as Witten-Bell interpolation or the Good-turing method [Wu and Zheng 2001] to estimate the probabilities of unseen events, this will be of no use when the size of training corpus is limited. From our observations, many the unseen events that occur when using N-gram LMs are unknown words or phrases. Such unknown words and phrases cannot be found in a dictionary. For example, the term “週休二 日” (two days off per week) is presently popular in Taiwan. We cannot find this term in a traditional dictionary. The term “週休二日” is a 4-word string pattern which consists of four words: “週” (a week), “休” (to rest), “二” (two), and …

Content truncated to 500 words