A Framework of Feature Selection Methods for Text Categorization

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 692–700, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP A Framework of Feature Selection Methods for Text Categorization Shoushan Li1 Rui Xia2 Chengqing Zong2 Chu-Ren Huang1 1 Department of Chinese and Bilingual Studies The Hong Kong Polytechnic University {shoushan.li,churenhuang} @gmail.com 2 National Laboratory of Pattern Recognition Institute of Automation Chinese Academy of Sciences {rxia,cqzong}@nlpr.ia.ac.cn Abstract In text categorization, feature selection (FS) is a strategy that aims at making text classifiers more efficient and accurate. However, when dealing with a new task, it is still difficult to quickly select a suitable one from various FS methods provided by many previous studies. In this paper, we propose a theoretic framework of FS methods based on two basic measurements: frequency measurement and ratio measurement. Then six popular FS methods are in detail discussed under this framework. Moreover, with the guidance of our theoretical analysis, we propose a novel method called weighed frequency and odds (WFO) that combines the two measurements with trained weights. The experimental results on data sets from both topic-based and sentiment classification tasks show that this new method is robust across different tasks and numbers of selected features. 1 Introduction With the rapid growth of online information, text classification, the task of assigning text documents to one or more predefined categories, has become one of the key tools for automatically handling and organizing text information. The problems of text classification normally involve the difficulty of extremely high dimensional feature space which sometimes makes learning algorithms intractable. A standard procedure to reduce the feature dimensionality is called feature selection (FS). Various FS methods, such as document frequency (DF), information gain (IG), mutual information (MI), χ2 -test (CHI), Bi-Normal Separation (BNS), and weighted log-likelihood ratio (WLLR), have been proposed for the tasks (Yang and Pedersen, 1997; Nigam et al., 2000; Forman, 2003) and make text classification more efficient and accurate. However, comparing these FS methods appears to be difficult because they are usually based on different theories or measurements. For example, MI and IG are based on information theory, while CHI is mainly based on the measurements of statistic independence. Previous comparisons of these methods have mainly depended on empirical studies that are heavily affected by the experimental sets. As a result, conclusions from those studies are sometimes inconsistent. In order to better understand the relationship between these methods, building a general theoretical framework provides a fascinating perspective. Furthermore, in real applications, selecting an appropriate FS method remains hard for a new task because too many FS methods are available due to the long history of FS studies. For example, merely in an early survey paper (Sebastiani, 2002), eight methods are mentioned. These methods are provided by previous work for dealing with different text classification tasks but none of them is shown to be robust across different classification applications. two In this paper, we propose a framework with basic measurements for theoretical comparison of …

Content truncated to 500 words