Information Retrieval Oriented Word Segmentation based on Character Association Strength Ranking

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1061–1069, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Information Retrieval Oriented Word Segmentation based on Character Associative Strength Ranking Yixuan Liu, Bin Wang, Fan Ding, Sheng Xu Information Retrieval Group Center for Advanced Computing Research Institute of Computing Technology Chinese Academy of Sciences Beijing, 100190, P.R.China {liuyixuan, wangbin, dingfan, xusheng}@ict.ac.cn Abstract This paper presents a novel, ranking-style word segmentation approach, called RSVM- Seg, which is well tailored to Chinese informa- tion retrieval(CIR). This strategy makes seg- mentation decision based on the ranking of the internal associative strength between each pair of adjacent characters of the sentence. On the training corpus composed of query items, a ranking model is learned by a widely-used tool Ranking SVM, with some useful statistical features, such as mutual information, differ- ence of t-test, frequency and dictionary infor- mation. Experimental results show that, this method is able to eliminate overlapping am- biguity much more effectively, compared to the current word segmentation methods. Fur- thermore, as this strategy naturally generates segmentation results with different granular- ity, the performance of CIR systems is im- proved and achieves the state of the art. 1 Introduction To improve information retrieval systems’ perfor- mance, it is important to comprehend both queries and corpus precisely. Unlike English and other western languages, Chinese does not delimit words by white-space. Word segmentation is therefore a key preprocessor for Chinese information retrieval to comprehend sentences. Due to the characteristics of Chinese, two main problems remain unresolved in word segmentation: segmentation ambiguity and unknown words, which are also demonstrated to affect the performance of Chinese information retrieval (Foo and Li, 2004). Overlapping ambiguity and combinatory ambiguity are two forms of segmentation ambiguity. The first one refers to that ABC can be segmented into AB C or A BC. The second one refers to that string AB can be a word, or A can be a word and B can be a word. In CIR, the combinatory ambiguity is also called segmentation granularity problem (Fan et al., 2007). There are many researches on the relationship between word segmentation and Chi- nese information retrieval (Foo and Li, 2004; Peng et al., 2002a; Peng et al., 2002b; Jin and Wong, 2002). Their studies show that the segmentation accuracy does not monotonically influence subse- quent retrieval performance. Especially the overlap- ping ambiguity, as shown in experiments of (Wang, 2006), will cause more performance decrement of CIR. Thus a CIR system with a word segmenter bet- ter solving the overlapping ambiguity, may achieve better performance. Besides, it also showed that the precision of new word identification was more im- portant than the recall. There are some researches show that when com- pound words are split into smaller constituents, bet- ter retrieval results can be achieved (Peng et al., 2002a). On the other hand, it is reasonable that the longer the word which co-exists in query and cor- pus, the more similarity they may have. A hypothe- sis, …

Content truncated to 500 words