Effective Tag Set Selection in Chinese Word Segmentation via Conditional Random Field Modeling

Effective Tag Set Selection in Chinese Word Segmentation via Conditional Random Field Modeling Hai Zhao1,2⋆, Chang-Ning Huang2, Mu Li2, and Bao-Liang Lu1 1 Department of Computer Science and Engineering, Shanghai Jiao Tong University, 1954 Hua Shan Rd., Shanghai 200030, China {zhaohai,blu}@cs.sjtu.edu.cn, 2 Microsoft Research Asia, 49, Zhichun Road, Haidian District, Beijing, China, 100080 cnhuang@msrchina.research.microsoft.com, muli@microsoft.com Abstract. This paper is concerned with Chinese word segmentation, which is regarded as a character based tagging problem under conditional random field framework. It is different in our method that we consider both feature template selection and tag set selection, instead of feature template focused only method in existing work. Thus, there comes an empirical comparison study of perfor- mance among different tag sets in this paper. We show that there is a significant performance difference as different tag sets are selected. Based on the proposed method, our system gives the state-of-the-art performance. 1 Introduction Chinese text is written without natural delimiters, so word segmentation is an essential first step in Chinese language processing. In this aspect, Chinese is quite different from English in which sentences of words delimited by white spaces. Though it seems very simple, Chinese word segmentation is not a trivial problem. Actually, it has been active area of research in computational linguistics for almost 20 years and has drawn more and more attention in the Chinese language processing community. To accomplish such a task, various technologies are developed [1]. To give a comprehensive comparison of Chinese segmentation on common test cor- pora, two International Chinese Word Segmentation Bakeoffs were held in 2003 and 2005, and there were 12 and 23 participants respectively [4], [5]. In all of proposed methods, character based tagging method [2] quickly rose in two Bakeoffs as a remarkable one with state-of-the-art performance. As reported in [5], the results of Bakeoff-2005 shows a general trend to a decrease in error rates from 3.9% to 2.8% compared to the results of Bakeoff-2003. Especially, two participants, Ng and Tseng, gave the best results in almost all test corpora [6], [7]. We continue to improve CRF-based tagging method of Chinese word segmentation on the track of Ng and Tseng in this study. It is different in our method that we consider ⋆ This work was finished while the first author visited Microsoft Research Asia 87 both feature template selection and tag set selection, instead of feature template focused only methods in previous work. That is, feature template selection was the main work if it was not the unique one before, while tag set is empirically specified aforehand. There are two kinds of test schemes in Chinese word segmentation Bakeoff, open and closed test. In the open test participants are allowed to use training data and any other linguistic resource including other training corpora, proprietary dictionaries and so forth. In the closed test only training data are allowed to used for the particular corpus. No other data is allowed. In this study, we will limit our comparison in closed test because additional …

Content truncated to 500 words