Reducing Weight Undertraining in Structured Discriminative Learning

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 89–95, New York, June 2006. c⃝2006 Association for Computational Linguistics Reducing Weight Undertraining in Structured Discriminative Learning Charles Sutton, Michael Sindelar, and Andrew McCallum Department of Computer Science University of Massachusetts Amherst Amherst, MA 01003 USA {casutton,mccallum}@cs.umass.edu, msindela@student.umass.edu Abstract Discriminative probabilistic models are very popular in NLP because of the latitude they afford in designing features. But training involves complex trade-offs among weights, which can be dangerous: a few highly- indicative features can swamp the contribution of many individually weaker features, causing their weights to be undertrained. Such a model is less robust, for the highly-indicative features may be noisy or missing in the test data. To ameliorate this weight undertraining, we intro- duce several new feature bagging methods, in which separate models are trained on subsets of the original features, and combined using a mixture model or a product of experts. These methods include the logarithmic opinion pools used by Smith et al. (2005). We evaluate fea- ture bagging on linear-chain conditional ran- dom fields for two natural-language tasks. On both tasks, the feature-bagged CRF performs better than simply training a single CRF on all the features. 1 Introduction Discriminative methods for training probabilistic models have enjoyed wide popularity in natural language pro- cessing, such as in part-of-speech tagging (Toutanova et al., 2003), chunking (Sha and Pereira, 2003), named- entity recognition (Florian et al., 2003; Chieu and Ng, 2003), and most recently parsing (Taskar et al., 2004). A discriminative probabilistic model is trained to maxi- mize the conditional probability p(y|x) of output labels y given input variables x, as opposed to modeling the joint probability p(y, x), as in generative models such as the Naive Bayes classifier and hidden Markov models. The popularity of discriminative models stems from the great flexibility they allow in defining features: because the distribution over input features p(x) is not modeled, it can contain rich, highly overlapping features without making the model intractable for training and inference. In NLP, for example, useful features include word bi- grams and trigrams, prefixes and suffixes, membership in domain-specific lexicons, and information from semantic databases such as WordNet. It is not uncommon to have hundreds of thousands or even millions of features. But not all features, even ones that are carefully engi- neered, improve performance. Adding more features to a model can hurt its accuracy on unseen testing data. One well-known reason for this is overfitting: a model with more features has more capacity to fit chance regulari- ties in the training data. In this paper, however, we focus on another, more subtle effect: adding new features can cause existing ones to be underfit. Training of discrimi- native models, such as regularized logistic regression, in- volves complex trade-offs among weights. A few highly- indicative features can swamp the contribution of many individually weaker features, even if the weaker features, taken together, are just as indicative of the output. Such a model is …

Content truncated to 500 words