Integrating Surface and Abstract Features for Robust Cross-Domain Chinese Word Segmentation

Proceedings of COLING 2012: Technical Papers, pages 1653–1670, COLING 2012, Mumbai, December 2012. Integrating Surface and Abstract Features for Robust Cross-Domain Chinese Word Segmentation LI Xiaoqing1, WANG Kun1, ZONG Chengqing1 and SU Keh-Yih2 (1) National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China (2) Behavior Design Corporation, Taiwan {xqli, kunwang, cqzong}@nlpr.ia.ac.cn, kysu@bdc.com.tw ABSTRACT Current character-based approaches are not robust for cross domain Chinese word segmentation. In this paper, we alleviate this problem by deriving a novel enhanced character-based generative model with a new abstract aggregate candidate-feature, which indicates if the given candidate prefers the corresponding position-tag of the longest dictionary matching word. Since the distribution of the proposed feature is invariant across domains, our model thus possesses better generalization ability. Open tests on CIPS-SIGHAN-2010 show that the enhanced generative model achieves robust cross-domain performance for various OOV coverage rates and obtains the best performance on three out of four domains. The enhanced generative model is then further integrated with a discriminative model which also utilizes dictionary information. This integrated model is shown to be either superior or comparable to all other models reported in the literature on every domain of this task. KEYWORDS : Chinese Word Segmentation, Cross-Domain, Robust Feature, Utilize Dictionary 1653 1 Introduction As words are the basic units for text analysis, Chinese word segmentation (CWS) is critical for many Chinese NLP tasks such as parsing. However, current state-of-the-art character-based approaches fail to give robust performance for cross-domain tests (Gao and Vogel, 2010; Huang et al., 2010; Jiang and Dong, 2010; Wang et al., 2012, Sun et al., 2012), despite their acceptable performance for in-domain tests. For example, with the same PKU-News training corpus, the best system in SIGHAN-2005 (Emerson, 2005) achieved a high in-domain performance (96.9% in F-score) in open1 tests, while the best cross-domain performance for the medicine domain only achieved 93.8%2 in CIPS-SIGHAN-2010 (Zhao and Liu, 2010) open tests. Their poor performances result not only from the fact that out-domains have higher out-of- vocabulary (OOV) rates (please refer Table 1 in Section 3.1), but also from the fact that various domains frequently possess different tag distributions for the same character, especially for those out-domain OOV words (which are also technical terms most of times). For example, “酸” (acid) is a typical suffix of medical terms, such as “尿酸” (uric acid) and “氨基酸” (amino acid), and is frequently tagged as “E” in the medicine domain. However, it is usually a single-character word which means „sour‟ in general text, and should be tagged as “S”. As a result, those surface features (such as character n-grams) adopted in the approaches mentioned above have more difficulty in identifying the correct position-tag, which is a member of {Beginning, Middle, End, Singleton}(Xue, 2003), of a character within an OOV word in this case. Since many technical OOV words appear in the out-domain text, the performance thus degrades sharply. For instance, when we test the main stream character-based discriminative approach for medicine domain in CIPS-SIGHAN-2010 contest, 27.7% …

Content truncated to 500 words