Pruning False Unknown Words to Improve Chinese Word Segmentation

Pruning False Unknown Words to Improve Chinese Word Segmentation Chooi-Ling GOH and Masayuki ASAHARA and Yuji MATSUMOTO Graduate School of Information Science Nara Institute of Science and Technology, Japan 8916-5 Takayama, Ikoma, Nara 630-0192, Japan � ling-g,masayu-a,matsu � @is.naist.jp Abstract During the process of unknown word detection in Chinese word segmentation, many detected word candidates are invalid. These false unknown word candidates deteriorate the overall segmentation accuracy, as it will affect the segmentation accuracy of known words. Therefore, we propose to eliminate as many invalid word candidates as possible by a pruning process. Our experiments show that by cutting down the invalid unknown word candidates, we improve the segmentation accuracy of known words and hence that of the overall segmentation accuracy. 1 Introduction Since written Chinese texts do not use any markers such as spaces to indicate the word boundaries, word segmentation has become an essential task prior to any other Chinese language processing. There are two main problems in this task, segmentation ambiguities and unknown word occurrences. We can either solve these two problems in one single process or make them as two separate processes. Both approaches have pros and cons. If we make them as two separate processes, then we can focus on each problem independently. However, if two problems are solved in one single process, then the processing time may be shortened, but the computation will become more complicated. It is difficult to judge which approach is better. In this paper, we would like to compare the results by using only a single model for word segmentation (Goh et al., 2004) and by joining two models (unknown word model and disambiguation model). The extracted unknown words go through a pruning step, to eliminate those unlikely to be word candidates. Our experiments show that two processes for word segmentation perform better than only with a single process. 2 Previous Work Chinese word segmentation has been a research topic since long time ago. At first, people believed in rule based method, and tried to apply rules for word segmentation. However, as new words and new patterns can always be created, it became more and more complicated to maintain such a rule based system. Later, with the evolution of segmented corpora, it has become possible to apply machine learning based method to train a model for word segmentation. In year 2003, a competition for Chinese word segmentation was carried out in SIGHAN1 workshop to compare the accuracy of various methods (Sproat and Emerson, 2003). It used to be difficult to compare the accuracy of various systems because the experiments had been done on different corpora. Therefore, this bakeoff intended to standardize the training and testing corpora, so that a fair evaluation could be made. Along the history, some researches proposed solving ambiguity problem and detecting unknown word in one process (Xue and Converse, 2002; Asahara et al., 2003; Fu and Luke, 2004; Goh et al., 2004) 1A Special Interest Group of the Association of Computational Linguistics, http://www.sighan.org/. PACLIC 18, December …

Content truncated to 500 words