Covering Ambiguity Resolution in Chinese Word Segmentation Based on Contextual Information

Covering Ambiguity Resolution in Chinese Word Segmentation Based on Contextual Information Xiao LUO; Maosong SUN National Lab. of Intelligent Tech. and Systems Tsinghua University, Beijing, China, 100084 lkc-dcs@mail.tsinghua.edu.cn Benjamin K TSOU Language Information Sciences Research Centre City University of Hong Kong, Hong Kong rlbtsou@uxmail.cityu.edu.hk Abstract Covering ambiguity is one of the two basic types of ambiguities in Chinese word segmentation. We regard its resolution as equivalent to word sense disambiguation, and make use of the classical vector space model in information retrieval to formulate the contexts of ambiguous words. A variation form of TFIDF weighting is proposed and a Chinese thesaurus is additionally utilized to cope with data sparseness problem. We select 90 frequent cases of covering ambiguities as the target. The training set includes 77654 sentences, and the test set includes 19242 sentences. The experimental results showed that our model has achieved 96.58% accuracy, outperforming the original form of TFIDF weighting as well as another baseline model, the hidden Markov model. 1 Introduction In Chinese texts, there are no separators, such as spaces in English, to explicitly indicate boundaries between words. So word segmentation is regarded as the first step in Chinese information processing systems. Ambiguity is one of the two main obstacles in Chinese word segmentation (another is unknown word). There are two basic types of ambiguities in Chinese: overlapping ambiguity and covering ambiguity. The focus of this paper is on covering ambiguity. Covering ambiguity is defined as follows: Given a word w�W, W is a Chinese lexicon, if w is a concatenation of multiple words 1 w … n w (n ≥ 2), i w �W (i=1…n), and in addition, both the sequence w and the sequence 1 w … wn can be realized in some sentences, then we term w a “covering ambiguity” (We also refer to w an “ambiguous word” throughout the paper), meanwhile, term the sequence w and 1 w … n w the combined form and the separated form of the covering ambiguity w respectively. Observe “��”, a two-character string with covering ambiguity. Firstly, three words, i.e. “�” (middle), “�”(will) and “��”(lieutenant general), are involved in it. Secondly, both sequences can stand in real sentences. For examples: (1a) 1955���������� (1b) ���������������� Correct segmentations are: (1a) 1955� | � | � | �� | 1955 year he by award �� | �� | � lieutenant general military rank . (In 1955, he was awarded the military rank “lieutenant genera”.) (1b) �� | � | �� | �� information in society development | � | � | �� | �� | �� | � middle will play key role . (Information will play a key role in society development.) Covering ambiguities constitute about 10 percent of segmentation ambiguities in running Chinese texts (Liang, 1987). Their resolution is more difficult than that of overlapping ambiguities, -- it depends heavily on contexts, including syntactic, semantic and even pragmatic information. According to an evaluation held by the 863 High-Tech Project of China, the highest accuracy for covering ambiguities …

Content truncated to 500 words