Enhancing Domain Portability of Chinese Segmentation Model Using Chi-Square Statistics and Bootstrapping

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 789–798, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Enhancing domain portability of Chinese segmentation model using chi-square statistics and bootstrapping Baobao Chang, Dongxu Han Institute of Computational Linguistics, Peking University Key Laboratory of Computational Linguistics(Peking University), Ministry Education, China Beijing, 100871, P.R.China chbb@pku.edu.cn,hweibo@126.com Abstract Almost all Chinese language processing tasks involve word segmentation of the language input as their first steps, thus robust and reli- able segmentation techniques are always re- quired to make sure those tasks well- performed. In recent years, machine learning and sequence labeling models such as Condi- tional Random Fields (CRFs) are often used in segmenting Chinese texts. Compared with traditional lexicon-driven models, machine learned models achieve higher F-measure scores. But machine learned models heavily depend on training materials. Although they can effectively process texts from the same domain as the training texts, they perform relatively poorly when texts from new do- mains are to be processed. In this paper, we propose to use χ2 statistics when training an SVM-HMM based segmentation model to im- prove its ability to recall OOV words and then use bootstrapping strategies to maintain its ability to recall IV words. Experiments show the approach proposed in this paper enhances the domain portability of the Chinese word segmentation model and prevents drastic de- cline in performance when processing texts across domains. 1 Introduction Chinese word segmentation plays a fundamental role in Chinese language processing tasks, because almost all Chinese language processing tasks are assumed to work with segmented input. After in- tensive research for more than twenty years, the performance of Chinese segmentation made con- siderable progress. The bakeoff series hosted by the Chinese Information Processing Society (CIPS) and ACL SIGHAN shows that an F measure of 0.95 can be achieved in the closed test tracks, in which only specified training materials can be used in learning segmentation models1. Traditional word segmentation approaches are lexicon-driven (Liang, 1987) and assume prede- fined lexicons of Chinese words are available. Segmentation results are obtained by finding a best match between the input texts and the lexicons. Such lexicon-driven approaches can be rule-based, statistic-based or in some hybrid form. Xue (2003) proposed a novel way of segmenting Chinese texts, and it views the Chinese word seg- mentation task as a character tagging task. Accord- ing to Xue’s approach, no predefined Chinese lexicons are required; a tagging model is learned by using manually segmented training texts. The model is then used to assign each character a tag indicating the position of this character within a word. Xue’s approach has become the most popu- lar approach to Chinese word segmentation for its high performance and unified way of dealing with out-of-vocabulary (OOV) issues. Most segmenta- tion work began to follow this approach later. Ma- jor improvements in this line of research include: 1) More sophisticated learning models were intro- duced other than the maximum entropy model that Xue used, such as …

Content truncated to 500 words