Modeling Category Structures with a Kernel Function

Modeling Category Structures with a Kernel Function Hiroya Takamura Precision and Intelligence Laboratory Tokyo Institute of Technology 4259 Nagatsuta Midori-ku Yokohama, 226-8503 Japan takamura@pi.titech.ac.jp Yuji Matsumoto Department of Information Technology Nara Institute of Science and Technology 8516-9 Takayama Ikoma Nara, 630-0101 Japan matsu@is.aist-nara.ac.jp Hiroyasu Yamada School of Information Science Japan Advanced Institute of Science and Technology 1-1 Asahidai Tatsunokuchi Ishikawa, 923-1292 Japan h-yamada@jaist.ac.jp Abstract We propose one type of TOP (Tangent vector Of the Posterior log-odds) kernel and apply it to text categorization. In a number of categoriza- tion tasks including text categorization, nega- tive examples are usually more common than positive examples and there may be several dif- ferent types of negative examples. Therefore, we construct a TOP kernel, regarding the prob- abilistic model of negative examples as a mix- ture of several component models respectively corresponding to given categories. Since each component model of our mixture model is ex- pressed using a one-dimensional Gaussian-type function, the proposed kernel has an advantage in computational time. We also show that the computational advantage is shared by a more general class of models. In our experiments, the proposed kernel used with Support Vector Machines outperformed the linear kernel and the Fisher kernel based on the Probabilistic La- tent Semantic Indexing model. 1 Introduction Recently, Support Vector Machines (SVMs) have been actively studied because of their high generalization abil- ity (Vapnik, 1998). In the formulation of SVMs, func- tions which measure the similarity of two examples take an important role. These functions are called kernel func- tions. The usual dot-product of two vectors respectively corresponding to two examples is often used. Although some variants to the usual dot-product are sometimes used (for example, higher-order polynomial kernels and RBF kernels), the distribution of examples is not taken into account in such kernels. However, new types of kernels have more recently been proposed; they are based on the probability distri- bution of examples. One is Fisher kernels (Jaakkola and Haussler, 1998). The other is TOP (Tangent vector Of the Posterior log-odds) kernels (Tsuda et al., 2002). While Fisher kernels are constructed on the basis of a genera- tive model of data, TOP kernels are based on the class- posterior probability, that is, the probability that the pos- itive class occurs given an example. However, in order to use those kernels, we have to select a probabilistic model of data. The selection of a model will affect categoriza- tion result. The present paper provides one solution to this issue. Specifically, we proposed one type of TOP kernel, because it has been reported that TOP kernels per- form better than Fisher kernels in terms of categorization accuracy. We briefly explain our kernel. We focus on negative examples in binary classification. Negative examples are usually more common than positive examples. There may be several different types of negative examples. Fur- thermore, the categories of negative examples are some- times explicitly given (for example, the situation where we are given documents, each of which has one of three …

Content truncated to 500 words