A Pragmatic Approach for Classical Chinese Word Segmentation

A Pragmatic Approach for Classical Chinese Word Segmentation Shilei Huang, Jiangqin Wu College of Computer Science and Technology, Zhejiang University Yuquan Campus, Zhejiang University, Zheda Road, Hangzhou, Zhejiang, China {21621129, wujq}@ zju.edu.cn Abstract Word segmentation, a fundamental technology for lots of downstream applications, plays a significant role in Natural Language Processing, especially for those languages without explicit delimiters, like Chinese, Korean, Japanese and etc. Basically, word segmentation for modern Chinese is worked out to a certain extent. Nevertheless, Classical Chinese is largely neglected, mainly owing to its obsoleteness. One of the biggest problems for the researches of Classical Chinese word segmentation (CCWS) is lacking in standard large-scale shareable marked-up corpora, for the fact that the most excellent approaches, solving word segmentation, are based on machine learning or statistical methods which need quality-assured marked-up corpora. In this paper, we propose a pragmatic approach founded on the difference of t-score (dts) and Baidu Baike (the largest Chinese-language encyclopedia like Wikipedia) in order to deal with CCWS without any marked-up corpus. We extract candidate words as well as their corresponding frequency from the Twenty-Five Histories (Twenty-Four Histories and Draft History of Qing) to build a lexicon, and conduct segmentation experiments with it. The F- Score of our approach on the whole evaluation data set is 76.84%. Compared with traditional collocation-based methods, ours makes the segmentation more accurate. Keywords: Classical Chinese, Word Segmentation, Difference of T-score, Encyclopedia, Collocation 1. Introduction Till now, great achievements have been made in modern Chinese word segmentation (CWS) while there is little progress in Classical Chinese word segmentation (CCWS) for the obsoleteness of Classical Chinese. However, it’s worth studying it. As a fundamental technology, word segmentation is a prerequisite of making deep analyses on Classical Chinese literature. From the word frequency we can discover the transition of words which can explain some linguistic phenomena, historical facts, traditional folk cultures, social culture, geographical information and etc. Furthermore, we can extract entities like person, location, official title, date and so forth, to dig into the history and culture, rather than get superficial knowledge by basic search like string matching. Additionally, many downstream applications like knowledge graph and QA system need to be built from these entities. In recent years, many researchers have made analyses and explorations of unstructured or structured Classical Chinese literature (Xu, 2016; Ouyang, 2016; Liu et al., 2015; Lee and Wong, 2013; Li et al., 2012; Fang, Lo and Chinn, 2009). According to the estimate of experts, the number of the existing Chinese ancient books is about from 80,000 to 100,000. More than 40,000 books (around 4.8 billion Chinese Characters) have been converted into digital version (Ouyang, 2016). However, there are few marked- up corpora, let alone some of them being not shareable. Currently, the most known Classical Chinese corpora, which are marked up with POS (part of speech) tags and segmented into words, are Academia Sinica Ancient Chinese Corpus1, Sheffield Corpus of Chinese2 and CityU Treebank of Classical Chinese Poems3. Unfortunately, they are not enough to do …

Content truncated to 500 words