A Probabilistic Approach to Compound Noun Indexing in Korean Texts

A Probabilistic Approach to Compound Noun Indexing in Korean Texts Hyouk R. Park and Young S. Han and Kang H. Lee Korea R&D Information Center/KIST P.O. Box 122 YuSong Taejon, 305-600, Korea { hrpark, yshan, khlee} @stissbs.kordic.re.kr Key-Sun Choi Computer Science Department KAIST YuSong Taejon, 305-701, Korea kschoi(O)world, kaist, ac. kr Abstract In this paper we address the prob- lem of compound noun indexing that is about segmenting or decomposing compound nouns into promising index terms. Compound nouns as index terms that usually subscribe to specific no- tions tend to increase the precision of retrieval performance. The use of the component nouns of a compound noun as index terms, on the other hand, may improve the recall performance, but can decrease the precision. Our proposed method to handle com- pound nouns with a goal to increase the recall while preserving the preci- sion computes the relevance of the com- ponent nouns of a compound noun to the document content by comparing the document sets that are supported by the component nouns and the terms of the document. The operational content of a term is represented as the proba- bilistic distribution of the term over the document set. Experiments with a set of 1,000 docu- ments show that our method gains 33% increase of retrieval performance com- pared to the indexing method without compound noun analysis, and is as good as manual decomposition by human ex- perts. 1 Introduction Automatic indexing renders a form of document representation that visualizes the content of the document more explicitly. Indices that are care- fully chosen to represent a document will bring about the improvement of retrieval performance in accuracy and time efficiency. The potential of a candidate index is often judged on the basis of its discriminating power over a docmnent set as well as its linguistic significance in the document. Thus, a good index term should distinguish a cer- tain class of documents from the rest of the doc- uments and be relevant to the subject matters of the class of documents to be indexed by the term. In general, automatic indexing consists of the identification of index terms and the assignment of weights to the terms (Salton 1983). An index term can be either a simple noun or a compound noun composed of more than one sim- ple nouns. Compound nouns tend to carry more specific contextual information than simple nouns, thus they are likely to contribute to the retrieval precision. Compound nouns may contain useful simple nouns that usually refer general contexts, and thus will boost the recall of retrieval. Process- ing compound nouns is decomposing them into simple nonns and evaluating the simple nouns as potential index terms. In both identifying and evaluating index terms, compound nouns require a different strategy from that for simple nouns. The identification of compound nouns involves a certain degree of linguistic or statistical analysis that varies from simple stemming to morphologi- cal analysis (Fagan 1989). What makes it even more complicated to han- dle …

Content truncated to 500 words