Virtual Examples for Text Classification with Support Vector Machines

Virtual Examples for Text Classification with Support Vector Machines Manabu Sassano Fujitsu Laboratories Ltd. 4-1-1, Kamikodanaka, Nakahara-ku, Kawasaki 211-8588, Japan sassano@jp.fujitsu.com Abstract We explore how virtual examples (artifi- cially created examples) improve perfor- mance of text classification with Support Vector Machines (SVMs). We propose techniques to create virtual examples for text classification based on the assump- tion that the category of a document is un- changed even if a small number of words are added or deleted. We evaluate the pro- posed methods by Reuters-21758 test set collection. Experimental results show vir- tual examples improve the performance of text classification with SVMs, especially for small training sets. 1 Introduction Corpus-based supervised learning is now a stan- dard approach to achieve high-performance in nat- ural language processing. However, the weakness of supervised learning approach is to need an anno- tated corpus, the size of which is reasonably large. Even if we have a good supervised-learning method, we cannot get high-performance without an anno- tated corpus. The problem is that corpus annota- tion is labor intensive and very expensive. In or- der to overcome this, several methods are proposed, including minimally-supervised learning methods (e.g., (Yarowsky, 1995; Blum and Mitchell, 1998)), and active learning methods (e.g., (Thompson et al., 1999; Sassano, 2002)). The spirit behind these methods is to utilize precious labeled examples max- imally. Another method following the same spirit is one using virtual examples (artificially created exam- ples) generated from labeled examples. This method has been rarely discussed in natural language pro- cessing. In terms of active learning, Lewis and Gale (1994) mentioned the use of virtual examples in text classification. They did not, however, take forward this approach because it did not seem to be possi- ble that a classifier created virtual examples of doc- uments in natural language and then requested a hu- man teacher to label them. In the field of pattern recognition, some kind of virtual examples has been studied. The first re- port of methods using virtual examples with Sup- port Vector Machines (SVMs) is that of Sch¨olkopf et al. (1996), who demonstrated significant improve- ment of the accuracy in hand-written digit recogni- tion (Section 3). They created virtual examples from labeled examples based on prior knowledge of the task: slightly translated (e.g., 1 pixel shifted to the right) images have the same label (class) of the orig- inal image. Niyogi et al. (1998) also discussed the use of prior knowledge by creating virtual examples and thereby expanding the effective training set size. The purpose of this study is to explore the effec- tiveness of virtual examples in NLP, motivated by the results of Sch¨olkopf et al. (1996). To our knowl- edge, use of virtual examples in corpus-based NLP has never been studied so far. It is, however, im- portant to investigate this approach by which it is expected that we can alleviate the cost of corpus an- notation. In particular, we focus on virtual examples with Support Vector Machines, introduced by Vap- nik (1995). The …

Content truncated to 500 words