Customizable Segmentation of Morphologically Derived Words in Chinese

Computational Linguistics and Chinese Language Processing Vol. 8, No. 1, February 2003, pp. 1-28 1  The Association for Computational Linguistics and Chinese Language Processing Customizable Segmentation of Morphologically Derived Words in Chinese Andi Wu* Abstract The output of Chinese word segmentation can vary according to different linguistic definitions of words and different engineering requirements, and no single standard can satisfy all linguists and all computer applications. Most of the disagreements in language processing come from the segmentation of morphologically derived words (MDWs). This paper presents a system that can be conveniently customized to meet various user-defined standards in the segmentation of MDWs. In this system, all MDWs contain word trees where the root nodes correspond to maximal words and leaf nodes to minimal words. Each non-terminal node in the tree is associated with a resolution parameter which determines whether its daughters are to be displayed as a single word or separate words. Different outputs of segmentation can then be obtained from the different cuts of the tree, which are specified by the user through the different value combinations of those resolution parameters. We thus have a single system that can be customized to meet different segmentation specifications. Keywords: segmentation standards, morphologically derive words, customizable systems, word-internal structures 1. Introduction A written sentence in Chinese consists of a string of evenly spaced characters with no delimiters between the words1. In any word-based Chinese language processing2, therefore, segmenting each sentence into words is a prerequisite. However, due to some special linguistic properties of Chinese words, there is not a generally accepted standard that can be *Microsoft Research Address: 21062 NE 81st Street, Redmond, WA, USA E-mail: andiwu@microsoft.com Phone: (O) 1-425-706-0985 (H) 1-425-868-8075 1 See Sproat [2000] for a theoretical account of this orthographic convention. 2 Character-based processing is also possible and has performed well in certain applications. 2 Andi Wu used to unambiguously determine “wordhood” in every case. 3 While native speakers of Chinese are often able to agree on how to segment a string of characters into words, there are a substantial number of cases where no agreement can be reached [Sproat et al. 1996]. Besides, different natural language processing (NLP) applications may have different requirements that call for different definitions of words and different granularities of word segmentation. This presents a challenging problem for the development of annotated Chinese corpora that are expected to be useful for training multiple types of NLP systems. It is also a challenge to any Chinese word segmentation system that claims to be capable of supporting multiple user applications. In what follows, we will discuss this problem mainly from the viewpoint of NLP and propose a solution that we have implemented and evaluated in an existing Chinese NLP system4. In Section 2, we will look at the problem areas where disagreements among different standards are most likely to arise. We will identify the alternatives in each case, discuss the computational motivation behind each segmentation option, and suggest possible solutions. This section can be skipped …

Content truncated to 500 words