Optimizing to Arbitrary NLP Metrics using Ensemble Selection

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 539–546, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Optimizing to Arbitrary NLP Metrics using Ensemble Selection Art Munson, Claire Cardie, Rich Caruana Department of Computer Science Cornell University Ithaca, NY 14850 {mmunson, cardie, caruana}@cs.cornell.edu Abstract While there have been many successful applica- tions of machine learning methods to tasks in NLP, learning algorithms are not typically designed to optimize NLP performance metrics. This paper evaluates an ensemble selection framework de- signed to optimize arbitrary metrics and automate the process of algorithm selection and parameter tuning. We report the results of experiments that in- stantiate the framework for three NLP tasks, using six learning algorithms, a wide variety of parame- terizations, and 15 performance metrics. Based on our results, we make recommendations for subse- quent machine-learning-based research for natural language learning. 1 Introduction Among the most successful natural language learn- ing techniques for a wide variety of linguistic phe- nomena are supervised inductive learning algo- rithms for classification. Because of their capa- bilities for accurate, robust, and efficient linguistic knowledge acquisition, they have been employed in many natural language processing (NLP) tasks. Unfortunately, supervised classification algo- rithms are typically designed to optimize accuracy (e.g. decision trees) or mean squared error (e.g. neu- ral networks). For many NLP tasks, however, these standard performance measures are inappropriate. For example, NLP data can be highly skewed in its distribution of positive and negative examples. In these situations, another metric (perhaps F-measure or a task-specific measure) that focuses on the per- formance of the minority cases is more appropriate. Indeed, as the NLP field matures more consideration will be given to evaluating the performance of NLP components in context (e.g. Is the system easy to use by end users? Does the component respect user pref- erences? How well does the entire system solve the specific problem?), leading to new and complicated metrics. Optimizing machine learning algorithms to arbitrary performance metrics, however, is not easily done. To exacerbate matters, the metric of interest might change depending on how the natural language learning (NLL) component is employed. Some ap- plications might need components with high re- call, for example; others, high precision or high F- measure or low root mean squared error. To obtain good results w.r.t. the new metric, however, a dif- ferent parameterization or different algorithm alto- gether might be called for, requiring re-training the classifier(s) from scratch. Caruana et al. (2004) have recently proposed en- semble selection as a technique for building an en- semble of classifiers that is optimized to an arbitrary performance metric. The approach trains a large number of classifiers using multiple algorithms and parameter settings, with the idea that at least some of the classifiers will perform well on any given per- formance measure. The best set of classifiers, w.r.t. the target metric, is then greedily selected. (Select- ing a set of size 1 is equivalent to parameter and algorithm tuning.) Like …

Content truncated to 500 words