A mixture-of-experts framework for text classification

A Mixture-of-Experts Framework for Text Classification Andrew Estabrooks Nathalie Japkowicz IBM Toronto Lab, Office 1B28B, SITE, University of Ottawa, 1150 Eglinton Avenue East, 150 Louis Pasteur, P.O. Box 450 Stn. A, North York, Ontario, Canada, M3C 1H7 Ottawa, Ontario, Canada K1N 6N5 aestabro@ca.ibm.com nat@site.uottawa.ca Abstract One of the particular characteristics of text classification tasks is that they present large class imbalances. Such a problem can easily be tackled using re- sampling methods. However, although these approaches are very simple to im- plement, tuning them most effectively is not an easy task. In particular, it is unclear whether oversampling is more effective than undersampling and which oversampling or undersampling rate should be used. This paper presents a method for combining different ex- pressions of the re-sampling approach in a mixture of experts framework. The proposed combination scheme is evalu- ated on a very imbalanced subset of the REUTERS-21578 text collection and is shown to be very effective on this do- main. 1 Introduction A typical use of Machine Learning methods in the context of Natural Language Processing is in the domain of text classification. Unfortunately, several characteristics specific to text data make its classification a difficult problem to handle. In particular, the data is typically highly dimensional and it presents a large class imbalance, i.e., there, typically, are very few documents on the topic of interest while texts on unrelated subjects abound. Furthermore, although large amounts of texts are available on line, little of them are labeled. Be- cause the class imbalance problem is known to negatively affect typical classifiers and because unlabeled data have no place in conventional su- pervised learning, using off-the-shelf supervised classifiers is likely not to be very successful in the context of text data. It is, instead, recom- mended to devise a classification method specifi- cally tuned to the text classification problem. The purpose of this study is to target some of the characteristics of text data in the hope of im- proving the effectiveness of the classification pro- cess. The topics of finding a good representa- tion for text data and dealing with its high di- mensionality have been investigated previously with, for example, the use of Wordnet [e.g., (Scott & Matwin, 1999)] and Support Vector Machines [e.g., (Joachims, 1998)], respectively. We will not be addressing these problems here. The question that we will tackle in this paper, instead, is that of dealing with the class imbalance, and, in the process of doing so, that of finding a way to take advantage of the extra, albeit, unlabeled data that are often left unused in classification studies.1 Several approaches have previously been pro- posed to deal with the class imbalance problem including a simple and yet quite effective method: re-sampling [e.g., (Lewis & Gale, 1994), (Kubat & Matwin, 1997), (Domingos, 1999)]. This paper deals with the two different types of re-sampling approaches: methods that oversample the small class in order to make it reach a size close to that of the larger class and methods that …

Content truncated to 500 words