Term Contributed Boundary Feature using Conditional Random Fields for Chinese Word Segmentation Task

Term Contributed Boundary Feature using Conditional Random Fields for Chinese Word Segmentation Task Tian-Jian Jiang†‡ Shih-Hung Liu*‡ Cheng-Lung Sung*‡ Wen-Lian Hsu†‡ †Department of Computer Science, National Tsing-Hua University *Department of Electrical Engineering, National Taiwan University ‡Institute of Information Science, Academia Sinica {tmjiang,journey,clsung,hsu}@iis.sinica.edu.tw Abstract This paper proposes a novel feature for conditional random field (CRF) model in Chinese word segmentation system. The system uses a conditional random field as machine learning model with one simple feature called term contributed boundaries (TCB) in addition to the “BIEO” character-based label scheme. TCB can be extracted from unlabeled corpora automatically, and segmentation variations of different domains are expected to be reflected implicitly. The dataset used in this paper is the closed training task in CIPS-SIGHAN-2010 bakeoff, including simplified and traditional Chinese texts. The experiment result shows that TCB does improve “BIEO” tagging domain-independently about 1% of the F1 measure score. Keywords: Term contributed boundary, Conditional Random fields, Chinese word segmentation. 1. Introduction Word segmentation is a trivial problem for most Western language, since there are clear delimiters (e.g. spaces) for individual words. However, for some Asia languages such as Chinese, Japanese and other language do not have word delimiters, word segmentation problem will be encountered if we want to do some further language processing, e.g. information retrieval, summarization and so on. Thus, the Chinese word segmentation could be viewed as a fundamental problem for natural language processing. Chinese word segmentation is still a challenging issue, and there is contest held in SIGHAN community [1]. The CIPS-SIGHAN-2010 bakeoff task of Chinese word segmentation is focused on cross-domain texts [2]. The design of data set is challenging particularly. The domain-specific training corpora remain unlabeled, and two of the test corpora keep domains unknown before releasing, therefore it is not easy to apply ordinary machine learning approaches, especially for the closed training evaluations. 143 Proceedings of the 22nd Conference on Computational Linguistics and Speech Processing (ROCLING 2010), Pages 143-156, Puli, Nantou,Taiwan, September 2010. Traditional approach for Chinese word segmentation problem is adopted dictionary along with lots of rules to segment the unlabelled texts [3]. Recent years, the statistical machine learning models, such as Hidden Markov Model (HMM) [4], Maximum Entropy Markov Model (MEMM) [5] and Conditional Random Field (CRF) [6], show the moderate performance for sequential labeling problem, especially CRF achieves better outcome. In this paper we propose a novel feature named term contributed boundary (TCB) for CRF model training. Since term contributed boundary extraction [10] is unsupervised, it is suitable for closed training task that any external resource or extra knowledge is not allowed. Without proper knowledge, the closed task of word segmentation can be hard when out-of-vocabulary (OOV) sequences occurred, where TCB extracted from test data directly may help. We also compare different character based label scheme “BI”, “BIO” and “BIEO” for model training. “B,” “I,” “E” and “O” mean the beginning of word, the internal of word, the end of word and the single character word, respectively. The character-based “BIO” tagging of Conditional Random …

Content truncated to 500 words