Overfitting Avoidance for Stochastic Modeling of Attribute-Value Grammars

In: Proceedings o/CoNLL-2000 and LLL-2000, pages 49-54, Lisbon, Portugal, 2000. Overfitting Avoidance for Stochastic Modeling of Attribute-Value Grammars Tony Mullen Alfa-Informatica University of Groningen mullen@let, rug. nl Miles Osborne Division of Informatics University of Edinburgh osborne@cogsci, ed. ac. uk Abstract We present a novel approach to the problem of overfitting in the training of stochastic mod- els for selecting parses generated by attribute- valued grammars. In this approach, statistical features are merged according to the frequency of linguistic elements within the features. The resulting models are more general than the orig- inal models, and contain fewer parameters. Em- pirical results from the task of parse selection suggest that the improvement in performance over repeated iterations of iterative scaling is more reliable with such generalized models than with ungeneralized models. 1 Introduction The maximum entropy technique of statistical modeling using random fields has proved to be an effective way of dealing with a variety of lin- guistic phenomena, in particular where mod- eling of attribute-valued grammars (AVG's) is concerned (Abney, 1997). This is largely be- cause its capacity for considering overlapping information sources allows the most to be made of situations where data is sparse. Neverthe- less, it is important that the statistical features employed be appropriate to the job. If the infor- mation contributed by the features is too spe- cific to the training data, overfitting becomes a problem (Chen and Rosenfeld, 1999; Osborne, 2000). In this event, a peak in model perfor- mance will be reached early on, and continued training yields progressive deterioration in per- formance. From a theoretical standpoint, over- fitting indicates that the model distribution is unrepresentative of the actual probabilities. In practice, it makes the performance of the model dependent upon early stopping of training. The point at which this must be done is not always reliably predictable. This paper describes an approach to feature selection for maximum entropy models which reduces the effects of overfitting. Candidate fea- tures are built up from basic grammatical el- ements found in the corpus. This "composi- tional" quality of the features is exploited for the purpose of overfitting reduction by means of ]eature merging. In this process, features which are similar to each other, save for cer- tain elements, are merged; i.e, their disjunc- tion is considered as a feature in itself, thus reducing the number of features in the model. The motivation behind this methodology is sim- ilar to that behind that of Kohavi and John (1997), but rather than seeking a proper subset of the candidate feature set, the merging pro- cedure attempts to compress the feature set, diminishing both noise and redundancy. The method differs from a simple feature cutoff, such as that described in Ratnaparkhi (1998), in that the feature cutoff eliminates statistical fea- tures directly, whereas the merging procedure attempts to generalize them. The method em- ployed here also derives inspiration from the no- tion of Bayesian model merging introduced by Stolcke and Omohundro (1994). Section 2 describes parse selection and dis- cusses …

Content truncated to 500 words