Using Mutual Information to Identify New Features for Text documents of Various Domains

Using Mutual Information to Identify New Features for Text documents of Various Domains GUO Zhi Li IBM China Research Lab 2/F, Haohai Building, No. 7, Shangdi 5th Street, Haidian District Beijing 100085, China guozhili@cn.ibm.com Abstract The task of identifying proper names, unknown words and new terms, is an important step in text processing systems. This paper describes a method of using mutual information to collect possible segments as candidates of these three feature types in a document scope. Then the construction and context of each possible feature is examined to determine its type, canonical form and meaning. Adding very little domain-specific knowledge, this method adapts to various domains easily. 1� Introduction Most text processing systems, e.g., information extraction, text categorization, clustering, and machine translation, use words (rather than characters) as basic units to build their algorithms. Thus morphological process like segmentation and feature identification becomes an important step, extremely for languages like Chinese, which lacks morphological marks like space separator between words or capital letters at proper names. The quality of segmentation and feature identification greatly influences the performance of the overall text processing systems. Unknown words, i.e., words used in a document but not collected in a segmentation dictionary, and unknown proper names (persons, locations, organizations and their abbreviations) often reduce the precision and quality of a Chinese segmentation algorithm. Although there are ways to collect a balanced word list as the basis of a dictionary, a document always has new words and new names. Hence there must be an efficient way to identify these features. If they are not correctly identified, the real single-character words and the lone characters that actually combine into a new word, the common words and proper names will all be mixed together, which impediments followed processing steps. This paper presents a method of using document-scope mutual information to identify three types of features in a unified algorithm. These three types of features are: • proper names, including sub-categories of person names, place names, organization names, and their abbreviation forms; • unknown words, including common words used in a document but not collected in segmentation dictionary, product and brand names; • document terms, usually some phrases formed by multiple words. These are often key concepts of a document. Mutual information is used to evaluate the coherence level of two consecutive characters and words in a document, and those bigrams of higher mutual information are assumed as "seed" of possible features. Then these seeds are extended to both their right and left sides, still using mutual information as a criterion to determine how long to be extended, to form a list of candidate features. The sifted patterns are assigned a category type and a confidence value, according to their internal constructions, their contexts, and their distribution in the document scope. Since the method uses document-scope statistics as a major criterion to identify proper names, unknown words, and document terms, it's relatively easy to adjust to various domains, by adding 372 relatively little domain-specific knowledge represented …

Content truncated to 500 words