Which Is Essential for Chinese Word Segmentation: Character versus Word

Which Is Essential for Chinese Word Segmentation: Character versus Word Chang-Ning Huang and Hai Zhao Microsoft Research Asia, 49, Zhichun Road, Haidian District, Beijing, China, 100080 {cnhuang, f-hzhao}@msrchina.research.microsoft.com Abstract. This paper proposes an empirical comparison between word-based method and character-based method for Chinese word segmentation. In three Chinese word segmentation Bakeoffs, character-based method quickly rose as a mainstream technique in this field. We disclose the linguistic background and sta- tistical feature behind this observation. Also, an empirical study between word- based method and character-based method are performed. Our results show that character-based method alone can work well for Chinese word segmentation without additional explicit word information from training corpus. 1 Introduction Chinese text is written without natural delimiters, so word segmentation is an essential first step in Chinese language processing. In this aspect, Chinese is quite different from English in which sentences of words delimited by white spaces. Though it seems very simple, Chinese word segmentation (CWS) is not a trivial problem. Actually, it has been active area of research in computational linguistics for almost 20 years and has drawn more and more attention in the Chinese language processing community. To accomplish such a task, various technologies are developed [1][2]. In the early work of Chinese word segmentation, word-based method once played the dominant role, in which maximum matching algorithm is the most typical method. Here, the term, word, means those known words are shown in known lexicon or training corpus (also are called in-vocabulary(IV) words.). Explicit known word information was still important learning object even after statistical methods were introduced in CWS [1]. To give a comprehensive comparison of Chinese segmentation on common test cor- pora, three International Chinese Word Segmentation Bakeoffs were held in 2003, 2005, and 20061, and there were 12, 23 and 23 participants, respectively [3], [4], [5]. Four seg- mentation corpora were presented in each Bakeoff. Thus, twelve corpora are available from Bakeoff 2003, 2005, and 2006. A summary of these corpora is shown in Table 1. In all of proposed methods, character-based tagging method [6], instead of tradi- tional word-based one, quickly rose in Bakeoff-2005 as a remarkable one with state-of- the-art performance. Especially, two participants, Ng and Tseng, gave the best results 1 In 2006, the name of the third Bakeoff has been changed into International Chinese Language Processing Bakeoff for the reason that named entity recognition task was added 1 in almost all tracks [7], [8]. In Bakeoff-2006, all participants whose system perfor- mance ranked first in a track at least used character-based method. Researchers turned to character-based method from traditional word-based method only with four years. The success of Bakeoffs not only gave some public consistent segmentation stan- dards, but also proposed a corpus-based segmentation standard representation, instead of the representation of known word lexicon and segmentation manual before. Thus Chinese word segmentation becomes more like corpus-based machine learning proce- dure in this sense. With the supply of common segmentation standards of Bakeoffs, the comparison problem on word-based method and character-based …

Content truncated to 500 words