A Comparison of Event Models for Naive Bayes Anti-Spam E-Mail Filtering

A Comparison of Event Models for Naive Bayes Anti-Spam E-Mail Filtering Karl-Michael Schneider University of Passau Department of General Linguistics Innstr. 40, D-94032 Passau schneide@phil.uni—passau.de Abstract We describe experiments with a Naive Bayes text classifier in the context of anti- spam E-mail filtering, using two different statistical event models: a mul- ti-variate Bernoulli model and a multi- nomial model. We introduce a family of feature ranking functions for feature se- lection in the multinomial event model that take account of the word frequency information. We present evaluation re- sults on two publicly available corpora of legitimate and spam E-mails. We find that the multinomial model is less biased towards one class and achieves slightly higher accuracy than the multi-variate Bernoulli model. 1 Introduction Text categorization is the task of assigning a text document to one of several predefined categories. Text categorization plays an important role in nat- ural language processing (NLP) and information retrieval (IR) applications. One particular applica- tion of text categorization is anti-spam E-mail fil- tering, where the goal is to block unsolicited mes- sages with commercial or pornographic content (UCE, spam) from a user's E-mail stream, while letting other (legitimate) messages pass. Here, the task is to assign a message to one of two cate- gories, legitimate and spam, based on the mes- sage's content. In recent years, a growing body of research has applied machine learning techniques to text cat- egorization and (anti-spam) E-mail filtering, in- cluding rule learning (Cohen, 1996), Naive Bayes (Sahami et al., 1998; Androutsopoulos et al., 2000b; Rennie, 2000), memory based learning (Androutsopoulos et al., 2000b), decision trees (Carreras and Marquez, 2001), support vector ma- chines (Drucker et al., 1999) or combinations of different learners (Sakkis et al., 2001). In these ap- proaches a classifier is learned from training data rather than constructed by hand, which results in better and more robust classifiers. The Naive Bayes classifier has been found par- ticularly attractive for the task of text categoriza- tion because it performs surprisingly well in many application areas despite its simplicity (Lewis, 1998). Bayesian classifiers are based on a prob- abilistic model of text generation. A text is gener- ated by first choosing a class according to some prior probability and then generating a text ac- cording to a class-specific distribution. The model parameters are estimated from training examples that have been annotated with their correct class. Given a new document, the classifier outputs the class which is most likely to have generated the document. From a linguistic point of view, a document is made up of words, and the semantics of the doc- ument is determined by the meaning of the words and the linguistic structure of the document. The Naive Bayesian classifier makes the simplifying assumption that the probability that a document is generated in some class depends only on the prob- abilities of the words given the context of the class, and that the words in a document are independent of each other. This is called the Naive …

Content truncated to 500 words