Improved Source-Channel Models for Chinese Word Segmentation

Improved Source-Channel Models for Chinese Word Segmentation1 Jianfeng Gao, Mu Li and Chang-Ning Huang Microsoft Research, Asia Beijing 100080, China {jfgao, t-muli, cnhuang}@microsoft.com 1 We would like to thank Ashley Chang, Jian-Yun Nie, Andi Wu and Ming Zhou for many useful discussions, and for comments on earlier versions of this paper. We would also like to thank Xiaoshan Fang, Jianfeng Li, Wenfeng Yang and Xiaodan Zhu for their help with evaluating our system. Abstract This paper presents a Chinese word segmen- tation system that uses improved source- channel models of Chinese sentence genera- tion. Chinese words are defined as one of the following four types: lexicon words, mor- phologically derived words, factoids, and named entities. Our system provides a unified approach to the four fundamental features of word-level Chinese language processing: (1) word segmentation, (2) morphological analy- sis, (3) factoid detection, and (4) named entity recognition. The performance of the system is evaluated on a manually annotated test set, and is also compared with several state-of- the-art systems, taking into account the fact that the definition of Chinese words often varies from system to system. 1 Introduction Chinese word segmentation is the initial step of many Chinese language processing tasks, and has attracted a lot of attention in the research commu- nity. It is a challenging problem due to the fact that there is no standard definition of Chinese words. In this paper, we define Chinese words as one of the following four types: entries in a lexicon, mor- phologically derived words, factoids, and named entities. We then present a Chinese word segmen- tation system which provides a solution to the four fundamental problems of word-level Chinese lan- guage processing: word segmentation, morpho- logical analysis, factoid detection, and named entity recognition (NER). There are no word boundaries in written Chinese text. Therefore, unlike English, it may not be de- sirable to separate the solution to word segmenta- tion from the solutions to the other three problems. Ideally, we would like to propose a unified ap- proach to all the four problems. The unified ap- proach we used in our system is based on the im- proved source-channel models of Chinese sentence generation, with two components: a source model and a set of channel models. The source model is used to estimate the generative probability of a word sequence, in which each word belongs to one word type. For each word type, a channel model is used to estimate the generative probability of a character string given the word type. So there are multiple channel models. We shall show in this paper that our models provide a statistical frame- work to corporate a wide variety linguistic knowl- edge and statistical models in a unified way. We evaluate the performance of our system us- ing an annotated test set. We also compare our system with several state-of-the-art systems, taking into account the fact that the definition of Chinese words often varies from system to system. In the rest of this paper: Section …

Content truncated to 500 words