An integrated approach for Chinese word segmentation

An integrated approach for Chinese word segmentation Guohong Fu Department of Linguistics The University of Hong Kong Pokfulam Road, Hong Kong glifu@hkucc.hku.hk K.K. Luke Department of Linguistics The University of Hong Kong Pokfulam Road, Hong Kong kiduke@hkusua.hku.hk ABSTRACT This paper presents an integrated approach for Chinese word segmentation, which can perform disambiguation and unknown word identification simultaneously on the input. In this work, a hybrid model is used to score known word candidates and unknown word candidates equally by incorporating the modified word-formation models (viz. word-juncture models and word- formation patterns) into word bigram models, with which different types of features are statistically computed and combined for this integrated segmentation, including internal word- formation power of components in a word, affinity relations between these components and the external contextual information. To enhance the precision and avoid the problem of combination explosion in word candidate construction, a filter algorithm is also given to block ineligible unknown word candidates. In this way, ambiguity and unknown word can be resolved effectively. The results of our experiment on Peking University corpus show that the integrated approach outperforms the other two-stage methods under discussion. 1 INTRODUCTION Chinese word segmentation aims to recognize the implicit word boundary delimiters in plain Chinese texts, which plays very important roles for most text-based applications, such as machine translation, information retrieval, text-t-speech synthesis and many more. During the past decades, many different techniques have been proposed for Chinese word segmentation, ranging from dictionary or rule based methods (Liang and Zheng 1991; Yeh and Lee 1991), statistical approaches (Fu and Wang 1999; Nie et al. 1995; Teahan et al. 2000; Wang et al. 2000; Yao 1997; Zhang et al. 2002), to machine learning approach (Hockenmaier and Brew 1998; Palmer 1997; Xue and Converse 2002). However, we are still faced with word boundary ambiguities and unknown words while developing a high-performance system for practical applications. What is more, ambiguity resolution and unknown word identification are often taken as two independent stages in previous word segmentation systems for Chinese. In this point, disambiguation is considered as a unique problem related to known word segmentation and unknown word identification is taken as a post-processing of known word segmentation. Although this two-stage strategy is simple and applicable, it usually fails to yield correct results for some complicated cases such as a mixture of ambiguities and unknown words. For instance, in a Chinese sentence 113 IT� , the correct segmentation for the fragment 11:1 4T-LEX should be 11:1 X/ (zhonglhang2 chang2ge2, Bank of China/Changgel). However, Most two-stage systems cannot yield this segmentation because this string has been wrongly segmented as /IT -K/ / (zhongl hang2zhang3 ge2, middlelpresidentlGe6 in process of known segmentation and there is no a mechanism to re-segment the word IT (hang2zhang3, president) during unknown word identification. Here, f7 (zhonglhang2, Bank of China) is an abbreviation of organization name, and (chang2ge2, Changge) is a place name. Recently, a variety of methods have been reported for this problem. Wu and Jiang (1998&2000) take word segmentation, …

Content truncated to 500 words