Lexicon Effects on Chinese Information Retrieval

Lexicon Effects on Chinese Information Retrieval K.L. Kwok Computer Science Dept., Queens College, City University of NY, Flushing, NY 11367, USA. kklqc @cunyvm.cuny.edu Abstract We investigate the effects of lexicon size and stopwords on Chinese information retrieval using our method of short-word segmentation based on simple language usage rules and statistics. These rules allow us to employ a small lexicon of only 2,175 entries and provide quite admirable retrieval results. It is noticed that accurate segmentation is not essential for good retrieval. Larger lexicons can lead to incremental improvements. The presence of stopwords do not contribute much noise to IR. Their removal risks elimination of crucial words in a query and adversely affect retrieval, especially when the queries are short. Short queries of a few words perform more than 10% worse than paragraph-size queries. 1 Introduction It is well known that a sentence in Chinese (or several other oriental languages) consists of a continuous string of 'characters' without delimiting white spaces to identify words. In Chinese, the characters are called ideographs. This makes it difficult to do machine studies on these languages since isolated words are needed for many purposes, such as linguistic analysis, machine translation, etc. Automatic methods for correctly isolating words in a sentence -- a process called word segmentation -- is therefore an important and necessary first step to be taken before other analysis can begin. Many researchers have proposed practical methods to resolve this problem such as (Nie et al., 1995, Wu and Tsang, 1995, Jin & Chen, 1996, Ponte & Croft, 1996, Sproat et al., 1996, Sun et al., 1997). Information retrieval (IR) deals with the problem of selecting relevant documents for a user need that is expressed in free text. The document collection is usually huge, of gigabyte size, and both queries and documents are domain unrestricted and unpredictable. When one does IR in the Chinese language with its peculiar property, then one would assume that accurate word segmentation is also a crucial first step before other processing can begin. However, in the recent 5th Text REtrieval Conference (TREC-5) where a fairly large scale Chinese IR experiment was performed [Kwok and Grunfeld, 199x], we have demonstrated that a simple word segmentation method, couple with a powerful retrieval algorithm, is sufficient to provide quite good retrieval results. Moreover, experiments by others using even simpler bigram representation of text (i.e. all consecutive overlapping two characters), both within and outside the TREC environment, also produce good results [Ballerini et al., 199x, Buckley et al., 199x, Chien, 1995, Liang et al., 1996]. This is a bit counter-intuitive because the bigram method leads to three times as large an indexing feature space compared with our segmentation (approximately 1.5 million vs 0.5 million), and one would expect that there are many random, non-content matchings between queries and documents that may adversely affect precision. Apparently, this is not so. Based on this observation, we made some adjustments to our lexicon, and provide some experimental results of the lexicon effects on retrieval …

Content truncated to 500 words