Document Classification Using a Finite Mixture Model

Document Classification Using a Finite Mixture Model Hang Li Kenji Yamanishi C&C Res. Labs., NEC 4-1-1 Miyazaki Miyamae-ku Kawasaki, 216, Japan Email: {lihang,yamanisi} @sbl.cl.nec.co.j p Abstract We propose a new method of classifying documents into categories. We define for each category a finite mixture model based on soft clustering of words. We treat the problem of classifying documents as that of conducting statistical hypothesis testing over finite mixture models, and employ the EM algorithm to efficiently estimate pa- rameters in a finite mixture model. Exper- imental results indicate that our method outperforms existing methods. 1 Introduction We are concerned here with the issue of classifying documents into categories. More precisely, we begin with a number of categories (e.g., 'tennis, soccer, skiing'), each already containing certain documents. Our goal is to determine into which categories newly given documents ought to be assigned, and to do so on the basis of the distribution of each document's words. 1 Many methods have been proposed to address this issue, and a number of them have proved to be quite effective (e.g.,(Apte, Damerau, and Weiss, 1994; Cohen and Singer, 1996; Lewis, 1992; Lewis and Ringuette, 1994; Lewis et al., 1996; Schutze, Hull, and Pedersen, 1995; Yang and Chute, 1994)). The simple method of conducting hypothesis testing over word-based distributions in categories (defined in Section 2) is not efficient in storage and suffers from the data sparseness problem, i.e., the number of parameters in the distributions is large and the data size is not sufficiently large for accurately es- timating them. In order to address this difficulty, (Guthrie, Walker, and Guthrie, 1994) have proposed using distributions based on what we refer to as hard 1A related issue is the retrieval, from a data base, of documents which are relevant to a given query (pseudo- document) (e.g.,(Deerwester et al., 1990; Fuhr, 1989; Robertson and Jones, 1976; Salton and McGill, 1983; Wong and Yao, 1989)). clustering of words, i.e., in which a word is assigned to a single cluster and words in the same cluster are treated uniformly. The use of hard clustering might, however, degrade classification results, since the dis- tributions it employs are not always precise enough for representing the differences between categories. We propose here to employ soft chsterinf, i.e., a word can be assigned to several different clusters and each cluster is characterized by a specific word probability distribution. We define for each cate- gory a finite mixture model, which is a linear com- bination of the word probability distributions of the clusters. We thereby treat the problem of classify- ing documents as that of conducting statistical hy- pothesis testing over finite mixture models. In or- der to accomplish hypothesis testing, we employ the EM algorithm to efficiently and approximately cal- culate from training data the maximum likelihood estimates of parameters in a finite mixture model. Our method overcomes the major drawbacks of the method using word-based distributions and the method based on hard clustering, while retaining their merits; it in fact includes those …

Content truncated to 500 words