A Character-Based Joint Model for Chinese Word Segmentation

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1173–1181, Beijing, August 2010 A Character-Based Joint Model for Chinese Word Segmentation Kun Wang and Chengqing Zong National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Science {kunwang,cqzong}@nlpr.ia.ac.cn Keh-Yih Su Behavior Design Corporation Kysu@bdc.com.tw Abstract The character-based tagging approach is a dominant technique for Chinese word segmentation, and both discrimi- native and generative models can be adopted in that framework. However, generative and discriminative charac- ter-based approaches are significantly different and complement each other. A simple joint model combining the character-based generative model and the discriminative one is thus proposed in this paper to take advantage of both approaches. Experiments on the Sec- ond SIGHAN Bakeoff show that this joint approach achieves 21% relative error reduction over the discriminative model and 14% over the generative one. In addition, closed tests also show that the proposed joint model outperforms all the existing approaches reported in the literature and achieves the best F- score in four out of five corpora. 1 Introduction Chinese word segmentation (CWS) plays an important role in most Chinese NLP applica- tions such as machine translation, information retrieval and question answering. Many statis- tical methods for CWS have been proposed in the last two decades, which can be classified as either word-based or character-based. The word-based approach regards the word as the basic unit, and the desired segmentation result is the best word sequence found by the search process. On the other hand, the character-based approach treats the word segmentation task as a character tagging problem. The final segmen- tation result is thus indirectly generated ac- cording to the tag assigned to each associated character. Since the vocabulary size of possible character-tag-pairs is limited, the character- based models can tolerate out-of-vocabulary (OOV) words and have become the dominant technique for CWS in recent years. On the other hand, statistical approaches can also be classified as either adopting a genera- tive model or adopting a discriminative model. The generative model learns the joint probabil- ity of the given input and its associated label sequence, while the discriminative model learns the posterior probability directly. Gen- erative models often do not perform well be- cause they make strong independence assump- tions between features and labels. However, (Toutanova, 2006) shows that generative mod- els can also achieve very similar or better per- formance than the corresponding discrimina- tive models if they have a structure that avoids unrealistic independence assumptions. In terms of the above dimensions, methods for CWS can be classified as: 1) The word-based generative model (Gao et al., 2003; Zhang et al., 2003), which is a well- known approach and has been used in many successful applications; 2) The word-based discriminative model (Zhang and Clark, 2007), which generates word candidates with both word and character features and is the only word-based model that adopts the discriminative approach； 3) The character-based discriminative model (Xue, 2003; Peng et al., 2004; Tseng et al., 2005; Jiang et al., 2008), …

Content truncated to 500 words