Chinese Word Segmentation by Mining Maximized Substrings

International Joint Conference on Natural Language Processing, pages 171–179, Nagoya, Japan, 14-18 October 2013. Chinese Word Segmentation by Mining Maximized Substrings Mo Shen, Daisuke Kawahara, and Sadao Kurohashi Graduate School of Informatics, Kyoto University Yoshida-honmachi, Sakyo-ku, Kyoto, 606-8501, Japan shen@nlp.ist.i.kyoto-u.ac.jp {dk,kuro}@i.kyoto-u.ac.jp Abstract A major problem in the field of Chinese word segmentation is the identification of out-of- vocabulary words. We propose a simple yet effective approach for extracting maximized substrings, which provide good estimations of unknown word boundaries. We also develop a new semi-supervised segmentation technique that incorporates retrieved substrings using discriminative learning. The effectiveness of this novel approach is demonstrated through experiments using both in-domain and out-of- domain data. 1. Introduction Chinese sentences are written without explicit word boundaries, which makes Chinese word segmentation (CWS) an initial and important step in Chinese language processing. Recent ad- vances in machine learning techniques have boosted the performance of CWS systems. On the other hand, a major difficulty in CWS is the problem of identifying out-of-vocabulary (OOV) words, as the Chinese language is continually and rapidly evolving, particularly with the rapid growth of the internet. A recent line of research to overcome this dif- ficulty is through exploiting characteristics of frequent substrings in unlabeled data. Statistical criteria for measuring the likelihood of a sub- string being a word have been proposed in previ- ous studies of unsupervised segmentation, such as accessor variety (Feng et al., 2004) and branching entropy (Jin and Tanaka-Ishii, 2006). This kind of criteria has been applied to enhance the performance of supervised segmentation sys- tems (Zhao and Kit, 2007; Zhao and Kit, 2008; Substring Freq 一致 3 界限数的期望值 2 一致认定界限 2 的期望值 3 认定界限数的 2 值 4 Table 1. A particular type of substrings with mul- tiple occurrences in the Chinese sentence: “使一致 认定界限数的期望值近似于一致正确界限数的期望 值，求得一致认定界限的期望值/认定界限数的 值。” Sun and Xu, 2011) by identifying unknown word boundaries. In this paper, instead of investigating statistical characteristics of batched substrings, we propose a novel method that extracts substrings as relia- ble word boundary estimations. The technique uses large-scale unlabeled data, and processes it on the fly. To illustrate the idea, we first consider the fol- lowing example taken from a scientific text: “使一致认定界限数的期望值近似于一致正确界限 数的期望值，求得一致认定界限的期望值/认定界 限数的值。” Without any knowledge of the Chinese lan- guage one may still notice that some substrings like “一致” and “的期望值”, occur multiple times in the sentence and are likely to be valid words or chains of words. Consider a particular type of frequent substring that cannot be simulta- neously extended by its surrounding characters while still being equal (Table 1). We can observe that the boundaries of such substrings can be used as perfect word delimiters. We can segment the sentence by simply treating the boundaries of each occurrence of a substring in Table 1 as word 171 陈[B-noun] 陈[S-verb] 陈[S-noun] 陈[B-verb] 答(answer) [verb] 记者(journalist) [noun] 问 (question) [noun] 记(record) [verb] 者(person) [noun] 问(ask) [verb] 德[B-noun] 德[B2-noun] 德[S-verb] 德[S-noun] 德[E-noun] 德[B-verb] 铭[B-noun] 铭[B2-noun] 铭[E-noun] 铭[S-noun] 铭[B3-noun] 铭[S-verb] 记[B-noun] 记[B2-noun] 记[M-noun] 记[S-noun] 记[B3-noun] 记[E-noun] 者[B-noun] 者[B2-noun] 者[M-noun] 者[S-noun] 者[B3-noun] …

Content truncated to 500 words