Feature Selection and Feature Extraction for Text Categorization

Feature Selection and Feature Extract ion for Text Categorization David D. Lewis Center for Information and Language Studies University of Chicago Chicago, IL 60637 ABSTRACT The effect of selecting varying numbers and kinds of fea- tures for use in predicting category membership was inves- tigated on the Reuters and MUC-3 text categorization data sets. Good categorization performance was achieved using a statistical classifier and a proportional assignment strat- egy. The optimal feature set size for word-based indexing was found to be surprisingly low (10 to 15 features) despite the large training sets. The extraction of new text features by syntactic analysis and feature clustering was investigated on the Reuters data set. Syntactic indexing phrases, clus- ters of these phrases, and clusters of words were all found to provide less effective representations than individual words. 1. Introduction Text categorization-the automated assigning of natural language texts to predefined categories based on their content-is a task of increasing importance. Its appli- cations include indexing texts to support document re- trieval [I], extracting data from texts [2], and aiding hu- mans in these tasks. The indexing language used to represent texts influences how easily and effectively a text categorization system can be built, whether the system is built by human engi- neering, statistical training, or a combination of the two. The simplest indexing languages are formed by treating each word as a feature. However, words have properties, such as synonymy and polysemy, that make them a less than ideal indexing language. These have motivated at- tempts to use more complex feature extraction methods in text retrieval and text categorization tasks. If a syntactic parse of text is available, then features can be defined by the presence of two or more words in particular syntactic relationships. We call such a fea- ture a syntactic indexing phrase. Another strategy is to use cluster analysis or other statistical methods to de- tect closely related features. Groups of such features can then, for instance, be replaced by a single feature corre- sponding to their logical or numeric sum. This strategy is referred to as term clustering. Syntactic phrase indexing and term clustering have op- posite effects on the properties of a text representation, which led us to investigate combining the two techniques [3]. However, the small size of standard text retrieval test collections, and the variety of approaches available for query interpretation, made it difficult to study purely representational issues in text retrieval experiments. In this paper we examine indexing language properties us- ing two text categorization data sets. We obtain much clearer results, as well as producing a new text catego- rization method capable of handing multiple, overlap- ping categories. 2. Data Sets and Tasks Our first data set was a set of 21,450 Reuters newswire stories from the year 1987 [4]. These stories have been manually indexed using 135 financial topic categories, to support document routing and retrieval. Particular care was taken in assigning categories [I]. All stories dated April 7, 1987 and earlier went into …

Content truncated to 500 words