An Improved Hierarchical Bayesian Model of Language for Document Classification

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 25–32 Manchester, August 2008 An Improved Hierarchical Bayesian Model of Language for Document Classification Ben Allison Department of Computer Science University of Sheffield UK ben@dcs.shef.ac.uk Abstract This paper addresses the fundamental problem of document classification, and we focus attention on classification prob- lems where the classes are mutually exclu- sive. In the course of the paper we advo- cate an approximate sampling distribution for word counts in documents, and demon- strate the model’s capacity to outperform both the simple multinomial and more re- cently proposed extensions on the classifi- cation task. We also compare the classi- fiers to a linear SVM, and show that pro- vided certain conditions are met, the new model allows performance which exceeds that of the SVM and attains amongst the very best published results on the News- groups classification task. 1 Introduction Document classification is one of the key technolo- gies in the emerging digital world: as the amount of textual information existing in electronic form increases exponentially, reliable automatic meth- ods to sift through the haystack and pluck out the occasional needle are almost a necessity. Previous comparative studies of different classi- fiers (for example, (Yang and Liu, 1999; Joachims, 1998; Rennie et al., 2003; Dumais et al., 1998)) have consistently shown linear Support Vector Ma- chines to be the most appropriate method. Gen- erative probabilistic classifiers, often represented by the multinomial classifier, have in these same © 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. studies performed poorly, and this empirical evi- dence has been bolstered by theoretical arguments (Lasserre et al., 2006). In this paper we revisit the theme of genera- tive classifiers for mutually exclusive classification problems, but consider classifiers employing more complex models of language; as a starting point we consider recent work (Madsen et al., 2005) which relaxes some of the multinomial assump- tions. We continue and expand upon the theme of that work, but identify some weaknesses both in its theoretical motivations and practical applica- tions. We demonstrate a new approximate model which overcomes some of these concerns, and demonstrate substantial improvements that such a model achieves on four classification tasks, three of which are standard and one of which is a newly created task. We also show the new model to be highly competitive to an SVM where the previous models are not. §2 of the paper describes previous work which has sought a probabilistic model of language and its application to document classification. §3 de- scribes the models we consider in this paper, and gives details of parameter estimation. §4 describes our evaluation of the models, and §5 presents the results of this evaluation. §6 explores reasons for the observed results, and finally §7 ends with some concluding remarks. 2 Related Work The problem of finding an appropriate and tractable model for language is one which has been studied in many different areas. In many cases, …

Content truncated to 500 words