The Measure of a Model

The Measure of a Model * Rebecca Brucet, danyce Wiebe~:, Ted Pedersent tDepartment of Computer Science and Engineering Southern Methodist University Dallas, TX 75275-0112 and :~Department of Computer Science New Mexico State University Los Cruces, NM 88003 rbruce~seas.smu.edu, wiebe@cs.nmsu.edu, pedersen@seas.smu.edu Abstract This paper describes measures for evaluating the three determinants of how well a probabilistic elas- sifter performs on a given test set. These determi- nants are the appropriateness, for the test set, of the results of (1)feature selection, (2) formulation of the parametric form of the model, and (3) pa- rameter estimation. These are part of any model formulation procedure, even if not broken out as separate steps, so the tradeoffs explored in this paper are relevant to a wide variety of methods. The measures are demonstrated in a large experi- ment, in which they are used to analyze the results of roughly 300 classifiers that perform word-sense disambiguation. Introduction This paper presents techniques that can be used to analyze the formulation of a probabilistic clas- sifter. As part of this presentation, we apply these techniques to the results of a large num- ber of classifiers, developed using the method- ology presented in (2), (3), (4), (5), (12) and (16), which tag words according to their meanings (i.e., that perform word-sense disambiguation). Other NLP tasks that have been performed using probabilistic classifiers include part-of-speech tag- ging (11), assignment of semantic classes (8), cue phrase identification (9), prepositional phrase at- tachment (15), other grammatical disambiguation tasks (6), anaphora resolution (7) and even trans- lation equivalence (1). In fact, it could be argued that any problem with a known set of possible so- lutions can be cast as a classification problem. A probabilistic classifier assigns, out of a set of possible classes, the one that is most probable according to a probabilistic model. The model ex- presses the relationships among the classification variable (the variable representing the classifica- tion tag) and var]ables that correspond to prop- *This research Was supported by the Office of Naval Research under grant number N00014-95-1-0776. erties of the ambiguous object and the context in which it occurs (the non-classification variables). Each model uniquely defines a classifier. The basic premise of a probabilistic approach to classification is that the process of assigning ob- ject classes is non-deterministic, i.e., there is no in- fallible indicator of the correct classification. The purpose of a probabilistic model is to characterize the uncertainty in the classification process. The probabilistie model defines, for each class and each ambiguous object, the probability that the object belongs to that class, given the values of the non- classification variables. The main steps in developing a probabilistic classifier and performing classification on the basis of a probability model are the following. 1 1. Feature Selection: selecting informative contextual features. These are the properties of the ambiguous object and the context in which it occurs that are indicative of its classification. Typically, each feature is represented as a random variable (a non-classification variable) in the prob- abilistic …

Content truncated to 500 words