An Empirical Comparison of Goodness Measures for Unsupervised Chinese Word Segmentation with a Unified Framework

An Empirical Comparison of Goodness Measures for Unsupervised Chinese Word Segmentation with a Unified Framework Hai Zhao∗and Chunyu Kit Department of Chinese, Translation and Linguistics, City University of Hong Kong, 83 Tat Chee Avenue, Kowloon, Hong Kong, China Email: haizhao@cityu.edu.hk, ctckit@cityu.edu.hk Abstract This paper reports our empirical evaluation and comparison of several popular good- ness measures for unsupervised segmenta- tion of Chinese texts using Bakeoff-3 data sets with a unified framework. Assuming no prior knowledge about Chinese, this frame- work relies on a goodness measure to iden- tify word candidates from unlabeled texts and then applies a generalized decoding al- gorithm to find the optimal segmentation of a sentence into such candidates with the greatest sum of goodness scores. Exper- iments show that description length gain outperforms other measures because of its strength for identifying short words. Further performance improvement is also reported, achieved by proper candidate pruning and by assemble segmentation to integrate the strengths of individual measures. 1 Introduction Unsupervised Chinese word segmentation was ex- plored in a number of previous works for various purposes and by various methods (Ge et al., 1999; Fu and Wang, 1999; Peng and Schuurmans, 2001; ∗The research described in this paper was supported by the Research Grants Council of Hong Kong S.A.R., China, through the CERG grant 9040861 (CityU 1318/03H) and by City Uni- versity of Hong Kong through the Strategic Research Grant 7002037. Dr. Hai Zhao was supported by a postdoctoral Re- search Fellowship in the Department of Chinese, Translation and Linguistics, City University of Hong Kong. Thanks four anonymous reviewers for their insightful comments! SUN et al., 2004; Jin and Tanaka-Ishii, 2006). How- ever, various heuristic rules are often involved in most existing works, and there has not been a com- prehensive comparison of their performance in a unified way with available large-scale “gold stan- dard” data sets, especially, multi-standard ones since Bakeoff-1 1. In this paper we will propose a unified frame- work for unsupervised segmentation of Chinese text. Four existing approaches to unsupervised segmenta- tions or word extraction are considered as its special cases, each with its own goodness measurement to quantify word likelihood. The output by each ap- proach will be evaluated using benchmark data sets of Bakeoff-32 (Levow, 2006). Note that unsuper- vised segmentation is different from, if not more complex than, word extraction, in that the former must carry out the segmentation task for a text, for which a segmentation (decoding) algorithm is indis- pensable, whereas the latter only acquires a word candidate list as output (Chang and Su, 1997; Zhang et al., 2000). 2 Generalized Framework We propose a generalized framework to unify the existing methods for unsupervised segmentation, as- suming the availability of a list of word candidates each associated with a goodness for how likely it is to be a true word. Let W = {{wi, g(wi)}i=1,...,n} be such a list, where wi is a word candidate and g(wi) 1First International Chinese Word Segmentation Bakeoff, at http://www.sighan.org/bakeoff2003 2The Third International Chinese Language …

Content truncated to 500 words