Comparative Experiments on Disambiguating Word Senses: An Illustration of the Role of Bias in Machine Learning

Comparative Experiments on Disambiguating Word Senses: An Illustration of the Role of Bias in Machine Learning Raymond J. Mooney Department of Computer Sciences University of Texas Austin, TX 78712-1188 mooney@cs.utexas.edu Abstract This paper describes an experimental compari- son of seven different learning algorithms on the problem of learning to disambiguate the meaning of a word from context. The algorithms tested include statistical, neural-network, decision-tree, rule-based, and case-based classification tech- niques. The specific problem tested involves dis- ambiguating six senses of the word "line" using the words in the current and proceeding sentence as context. The statistical and neural-network methods perform the best on this particular prob- lem and we discuss a potential reason for this ob- served difference. We also discuss the role of bias in machine learning and its importance in explain- ing performance differences observed on specific problems. Introduction Recent research in empirical (corpus-based) natu- ral language processing has explored a number of different methods for learning from data. Three general approaches are statistical, neural-network, and symbolic machine learning and numerous spe- cific methods have been developed under each of these paradigms (Wermter, Riloff, & Scheler, 1996; Charniak, 1993; Reilly & Sharkey, 1992). An important question is whether some methods perform significantly better than others on partic- ular types of problems. Unfortunately, there have been very few direct comparisons of alternative methods on identical test data. A somewhat indirect comparison of apply- ing stochastic context-free grammars (Periera & Shabes, 1992), a transformation-based method (Brill, 1993), and inductive logic program- ming (Zelle & Mooney, 1994) to parsing the ATIS (Airline Travel Information Service) cor- pus from the Penn Treebank (Marcus, Santorini, & Marcinkiewicz, 1993) indicates fairly similar performance for these three very different meth- ods. Also, comparisons of Bayesian, information- retrieval, neural-network, and case-based methods on word-sense disambiguation have also demon- strated similar performance (Leacock, Towell, & Voorhees, 1993b; Lehman, 1994). However, in a comparison of neural-network and decision-tree methods on learning to generate the past tense of an English verb, decision trees performed sig- nificantly better (Ling & Marinov, 1993; Ling, 1994). Subsequent experiments on this problem have demonstrated that an inductive logic pro- gramming method produces even better results than decision trees (Mooney & Califf, 1995). In this paper, we present direct comparisons of a fairly wide range of general learning algo- rithms on the problem of discriminating six senses of the word "line" from context, using data as- sembled by Leacock et al. (1993b). We compare a naive Bayesian classifier (Duda & Hart, 1973), a perceptron (Rosenblatt, 1962), a decision-tree learner (Quinlan, 1993), a k nearest-neighbor clas- sifier (Cover & Hart, 1967), logic-based DNF (dis- junctive normal form) and CNF (conjunctive nor- mal form) learners (Mooney, 1995) and a decision- list learner (Rivest, 1987). Tests on all methods used identical training and test sets, and ten sep- arate random trials were run in order to measure average performance and allow statistical testing of the significance of any observed differences. On this particular task, we found that …

Content truncated to 500 words