Language and Task Independent Text Categorization with Simple Language Models

Language and Task Independent Text Categorization with Simple Language Models Fuchun Peng Dale Schuurmans Shaojun Wang School of Computer Science, University of Waterloo 200 University Avenue West, Waterloo, Ontario, Canada, N2L 3G1 {f3peng, dale, sjwang}@cs.uwaterloo.ca Abstract We present a simple method for language inde- pendent and task independent text categoriza- tion learning, based on character-level n-gram language models. Our approach uses simple information theoretic principles and achieves effective performance across a variety of lan- guages and tasks without requiring feature se- lection or extensive pre-processing. To demon- strate the language and task independence of the proposed technique, we present experimen- tal results on several languages—Greek, En- glish, Chinese and Japanese—in several text categorization problems—language identifica- tion, authorship attribution, text genre classifi- cation, and topic detection. Our experimental results show that the simple approach achieves state of the art performance in each case. 1 Introduction Text categorization concerns the problem of automati- cally assigning given text passages (paragraphs or doc- uments) into predefined categories. Due to the rapid ex- plosion of texts in digital form, text categorization has become an important area of research owing to the need to automatically organize and index large text collections in various ways. Such techniques are currently being ap- plied in many areas, including language identification, authorship attribution (Stamatatos et al., 2000), text genre classification (Kesseler et al., 1997; Stamatatos et al., 2000), topic identification (Dumais et al., 1998; Lewis, 1992; McCallum, 1998; Yang, 1999), and subjective sen- timent classification (Turney, 2002). Many standard machine learning techniques have been applied to automated text categorization problems, such as naive-Bayes classifiers, support vector machines, lin- ear least squares models, neural networks, and K-nearest neighbor classifiers (Yang, 1999; Sebastiani, 2002). A common aspect of these approaches is that they treat text categorization as a standard classification problem, and thereby reduce the learning process to two simple steps: feature engineering, and classification learning over the feature space. Of these two steps, feature engineering is critical to achieving good performance in text categoriza- tion problems. Once good features are identified, almost any reasonable technique for learning a classifier seems to perform well (Scott, 1999). Unfortunately, the standard classification learning methodology has several drawbacks for text categoriza- tion. First, feature construction is usually language de- pendent. Various techniques such as stop-word removal or stemming require language specific knowledge to de- sign adequately. Moreover, whether one can use a purely word-level approach is itself a language dependent issue. In many Asian languages such as Chinese or Japanese, identifying words from character sequences is hard, and any word-based approach must suffer added complexity in coping with segmentation errors. Second, feature se- lection is task dependent. For example, tasks like au- thorship attribution or genre classification require atten- tion to linguistic style markers (Stamatatos et al., 2000), whereas topic detection systems rely more heavily on bag of words features. Third, there are an enormous num- ber of possible features to consider in text categorization problems, and standard feature selection approaches do not always cope …

Content truncated to 500 words