Using Bag-of-Concepts to Improve the Performance of Support Vector Machines in Text Categorization

Using Bag-of-Concepts to Improve the Performance of Support Vector Machines in Text Categorization Magnus Sahlgren SICS Box 1263 SE-164 29 Kista Sweden mange@sics.se Rickard C¨oster SICS Box 1263 SE-164 29 Kista Sweden rick@sics.se Abstract This paper investigates the use of concept- based representations for text categoriza- tion. We introduce a new approach to cre- ate concept-based text representations, and apply it to a standard text categorization collection. The representations are used as input to a Support Vector Machine classi- fier, and the results show that there are cer- tain categories for which concept-based rep- resentations constitute a viable supplement to word-based ones. We also demonstrate how the performance of the Support Vec- tor Machine can be improved by combining representations. 1 Introduction Text categorization is the task of assigning a text1 to one or more of a set of predefined cat- egories. As with most other natural language processing applications, representational factors are decisive for the performance of the catego- rization. The incomparably most common rep- resentational scheme in text categorization is the Bag-of-Words (BoW) approach, in which a text is represented as a vector ⃗t of word weights, such that ⃗ti = (w1...wn) where wn are the weights (usually a tf ×idf -value)2 of the words in the text. The BoW representation ignores all semantic or conceptual information; it simply looks at the surface word forms. There have been attempts at deriving more sophisticated representations for text catego- rization, including the use of n-grams or phrases 1In the remainder of this paper, we use the terms “text” and “document” synonymously. 2The tf ×idf measure is a standard weighting scheme, where tf i is simply the frequency of word i in the doc- ument, and idf is the inverse document frequency, given by N ni where N is the total number of documents in the data, and ni is the number of documents in which word i occurs. The most common version of the tf ×idf for- mula is wi = tf i ×log N ni (Baeza-Yates and Ribeiro-Neto, 1999). (Lewis, 1992; Dumais et al., 1998), or augment- ing the standard BoW approach with synonym clusters or latent dimensions (Baker and Mc- Callum, 1998; Cai and Hofmann, 2003). How- ever, none of the more elaborate representations manage to significantly outperform the stan- dard BoW approach (Sebastiani, 2002). In ad- dition to this, they are typically more expensive to compute. What interests us in this paper is the differ- ence between using standard BoW and more elaborate, concept-based representations. Since text categorization is normally cast as a prob- lem concerning the content of the text (Du- mais et al., 1998), one might assume that look- ing beyond the mere surface word forms should be beneficial for the text representations. We believe that, even though BoW representations are superior in most text categorization tasks, concept-based schemes do provide important in- formation, and that they can be used as a sup- plement to the BoW representations. Our goal is therefore to investigate whether …

Content truncated to 500 words