Feature Selection for Short Text Classification using Wavelet Packet Transform

Proceedings of the 19th Conference on Computational Language Learning, pages 321–326, Beijing, China, July 30-31, 2015. c⃝2015 Association for Computational Linguistics Feature Selection for Short Text Classification using Wavelet Packet Transform Anuj Mahajan Indian Institute of Technology, Delhi Hauz Khas, New Delhi 110016 India anujmahajan.iitd@gmail.com Sharmistha, Shourya Roy Xerox Research Centre India Bangalore - 560103 India {sharmistha.jat,shourya.roy}@xerox.com Abstract Text classification tasks suffer from curse of dimensionality due to large feature space. Short text data further exacerbates the prob- lem due to their sparse and noisy nature. Fea- ture selection thus becomes an important step in improving the classification performance. In this paper, we propose a novel feature se- lection method using Wavelet Packet Trans- form. Wavelet Packet Transform (WPT) has been used widely in various fields due to its efficiency in encoding transient signals. We demonstrate how short text classification task can be benefited by feature selection using WPT due to their sparse nature. Our technique chooses the most discriminating features by computing inter-class distances in the trans- formed space. We experimented extensively with several short text datasets. Compared to well known techniques our approach reduces the feature space size and improves the overall classification performance significantly in all the datasets. 1 Introduction Text classification task consists of assigning a docu- ment to one or more classes. This can be done us- ing machine learning techniques by training a model with labelled documents. Documents are usually repre- sented as vectors with a variety of techniques like bag- of-words(unigram, bigram), TFIDF representation, etc. Typically, text corpora have very high dimensional document representation equal to the size of vocabu- lary. This leads to curse of dimensionality1 in machine learning models, thereby degrading the performance. Short text corpora, like SMS, tweets, etc., in partic- ular suffer from sparse high dimensional feature space, due to large vocabulary and short document length. To give an idea as to how these factors affect the size of the feature space we compare Reuters with Twitter data corpus. In Reuters-21578 corpus there are approx- imately 2.5 Million words in total and 14506 unique vocabulary entries after standard preprocessing steps 1https://en.wikipedia.org/wiki/Curse_ of_dimensionality (which is the dimensionality of the feature space). However, Twitter 1 corpus, we used for experiments has approximately 15,000 words in total and feature space size of 7423 words. Additionally, the average length of an English tweet is 10-12 words whereas the average length of a document in Reuters-21578 news classification corpus is 200 words. Therefore, the di- mensionality is extremely high even for small corpora with short texts. In addition, the average number of words in a document is significantly less in short text data leading to higher sparsity of feature space repre- sentation of documents. Owing to this high dimensionality problem, one of the important steps in text classification workflows is feature selection. Feature selection techniques for tradi- tional documents have been aplenty and a few seminal survey articles have been written on this topic (Blitzer, 2008). In contrast, for short text there is much …

Content truncated to 500 words