Evaluating Text Categorization I

Evaluating Text Categorization I David D. Lewis Computer and Information Science Dept. University of Massachusetts Amherst, MA 01003 ABSTRACT While certain standard procedures are widely used for evalu- ating text retrieval systems and algorithms, the sarne is not true for text categorization. Omission of important data from reports is common and methods of measuring effectiveness vary widely. This ]'ms m~de judging the relative merits of techniques for text categorization dif~.cult and has disguised important research is- sues. In this paper I discuss a variety of ways of evaluating the effectiveness of text categorization systems, drawing both on re- ported categorization experiments and on methods used in eval- uating query-driven retrieval. I also consider the extent to which the same evaluation methods may be used with systems for text extraction, a more complex task. In evaluatlng either kind of system, the purpose for which the output is to be used is crucial in choosing appropriate evaluation methods. INTRODUCTION Text classification systems, i.e. systems which can make distinctions between meaningful classes of texts, have been widely studied in information retrieval and natural language processing. The majority of information retrieval research has been devoted to a particular form of text classification-- ~ext raft/eva/. Text retrieval systems find or route texts in response to arbitrary user queries or interest profiles. Eval- uation has been a focus of research in text retrieval since the beginning, and standard evaluation methods are in wide use. A smaller, but significant, body of work has examined a task variously known as machine-aided indexing, automated indexing, authority control, or text categorization. Text cat- egorization is the assignment of texts to one or more of a pre-existing set of categories, rather than classifying them in response to an arbitrary query. Categorization may be performed for a wide range of reasons, either as an end in itself or as a component of a larger system. i Current Address: Center for Information and Language Studies; University of Chicago; Chicago, IL 60637; le~ig@tira.uchicago.edu The literature on text categorization is widely scattered and shows little agreement on evaluation methods. This makes it very difficult to draw conclusions about the relative effectiveness of techniques so that, unlike the situation in query-driven retrieval, there is no consensus on a set of basic evaluation methods for text categorization. In this paper I discuss measures of effectiveness for text categorization systems and algorithms. Effectiveness refers to the ability of a categorization to supply information to a system or user that wants to access the texts. Measuring effectiveness is just one of several kinds of evaluation that should be considered [Spa81a, CH88, PFg0]. After considering effectiveness evaluation for text cate- gorization we will turn to a related task, text extraction, and consider what role the effectiveness measures discussed for categorization have there. A common theme is the need to consider in an evaluation the purpose for which informa- tion is generated from the text. I will have occasion in the following to repeatedly refer to a chapter by Tague [Tag81] …

Content truncated to 500 words