Chinese Word Segmentation as Character Tagging

Computational Linguistics and Chinese Language Processing Vol. 8, No. 1, February 2003, pp.29-48 29  The Association for Computational Linguistics and Chinese Language Processing Chinese Word Segmentation as Character Tagging Nianwen Xue* Abstract In this paper we report results of a supervised machine-learning approach to Chinese word segmentation. A maximum entropy tagger is trained on manually annotated data to automatically assign to Chinese characters, or hanzi, tags that indicate the position of a hanzi within a word. The tagged output is then converted into segmented text for evaluation. Preliminary results show that this approach is competitive against other supervised machine-learning segmenters reported in previous studies, achieving precision and recall rates of 95.01% and 94.94% respectively, trained on a 237K-word training set. Key Words: Chinese word segmentation, supervised machine-learning, maximum entropy, character tagging 1. Introduction It is generally agreed among researchers that word segmentation is a necessary first step in Chinese language processing. However, unlike English text in which sentences are sequences of words delimited by white spaces, in Chinese text, sentences are represented as strings of Chinese characters or hanzi without similar natural delimiters. Therefore, the first step in a Chinese language processing task is to identify the sequence of words in a sentence and mark boundaries in appropriate places. This may sound simple enough but in reality identifying words in Chinese is a non-trivial problem that has drawn a large body of research in the Chinese language processing community [Fan and Tsai, 1988; Gan, 1995; Gan, Palmer, and Lua, 1996; Guo, 1997; Jin and Chen, 1998; Sproat and Shih, 1990; Sproat et al., 1996; Wu and Jiang, 1998; Wu, 2003]. It is easy to demonstrate that the lack of natural delimiters itself is not the heart of the problem. In a hypothetical language where all words are represented with a finite set of symbols, if one subset of the symbols always start a word and another subset, mutually exclusive from the previous subset, always end a word, identifying words would be a trivial * Institute for Research in Cognitive Science , Suite 400A, 3401 Walnut Street University of Pennsylvania, Philadelphia, PA 19104, USA E-mail: xueniwen@linc.cis.upenn.edu 30 Nianwen Xue exercise. Nor can the problem be attributed to the lack of inflectional morphology. Although it is true in Indo-European languages inflectional affixes can generally be used to signal word boundaries, it is conceivable that a hypothetical language can use symbols other than inflectional morphemes to serve the same purpose. Therefore the issue is neither the lack of natural word delimiters nor the lack of inflectional morphemes in a language, rather it is whether the language has a way of unambiguously signaling the boundaries of a word. The real difficulty in automatic Chinese word segmentation is the lack of such unambiguous word boundary indicators. In fact, most hanzi can occur in different positions within different words. The examples in Table 1 show how the Chinese character 产 (“produce”) can occur in four different positions. This state of affairs makes it impossible to …

Content truncated to 500 words