Implementation and Evaluation of Scalable Approaches for Automatic Chinese Text Categorization

IMPLEMENTATION AND EVALUATION OF SCALABLE APPROACHES FOR AUTOMATIC CHINESE TEXT CATEGORIZATION Jyh-Jong Tsay , Jing-Doo Wang , Chun-Fu Pai and Ming-Kuen Tsay Department of Computer Science and Information Engineering National Chung Cheng University Chiayi, Taiwan 62107, ROC. {tsay, jdwang, pcf84, tmc861@cs.ccu.edu.tw Abstract The purpose of this research is to identify scalable approaches that can handle large amount of training data such as several years of news articles, and automatically assign predefined category to Chinese free text documents. Our approach consists of the following processes: (i) term extraction, (ii) term selection, and (iii) document classification. The approach first builds a recently developed SB-tree to identify all repeated substrings, called patterns, from the text. We then proceed to identify possible boundary of terms appearing in the identified patterns. After terms are extracted from the training articles, we run term selection algorithms to select the most significant terms and to reduce the number of terms to an acceptable level. The selected terms are med by the classifier to assign a predefined category to each text document. Our current experiment uses CNA one year news as training data, which consists of 73,420 articles and is far more than previous related research. In the experiment, we implement and compare four term selection methods, the odds ratio method, the mutual information method, the information gain method and the x2-test method, when they are combined with the naive Bayes classifier. Keywords: Text Categorization, Term Selection, Naive Bayes Classifier, Information Retrieval. 1 Introduction Text categorization is the problem of automatically assigning predefined categories to free text documents, and is gaining more and more importance as the amount of text data available on World Wide Web grows dramatically. A well classified text database will be very helpful for a user to identify interesting data from the huge collection of texts. There are many studies about the text categorization as well as web-page classification [11, 3, 7, 8, 21, 25, 26, 18, 6, 5, 2, 10]. While there are a great number of researches on automatic text categorization for English texts, text categorization for Asian languages such as Chinese, Japanese, Korean and Thai has not been studied seriously until recently [17, 29, 1]. It is well known that written Asian language consists of strings of ideograph separated by punctuation signs. An ideograph (or character) can function as a word with meaning(s), or it can act as an alphabet to form a "word" with one or more adjacent characters. Determining the boundaries of single or multi-character words in a string, a process called segmentation [4], is very difficult because no delimiter or while space is used in the text and one has to rely on the context -179- contents. Because text segmentation is not straightforward, 1-grams, 2-grams and n-grams have been used as indexing terms to represent documents in Asian languages. Among them, 1-gram- based approaches is the simplest one that uses single characters as indexing terms, and should be good for recall in information retrieval(IR) because it guarantees that if …

Content truncated to 500 words