Investigating GIS and Smoothing for Maximum Entropy Taggers

Investigating GIS and Smoothing for Maximum Entropy Taggers James R. Curran and Stephen Clark School of Informatics University of Edinburgh 2 Buccleuch Place, Edinburgh. EH8 9LW fjamesc,stephencl@cogsci.ed.ac.uk Abstract This paper investigates two elements of Maximum Entropy tagging: the use of a correction feature in the Gener- alised Iterative Scaling (Gis) estimation algorithm, and techniques for model smoothing. We show analytically and empirically that the correction feature, assumed to be required for the correct- ness of GIS, is unnecessary. We also ex- plore the use of a Gaussian prior and a simple cutoff for smoothing. The exper- iments are performed with two tagsets: the standard Penn Treebank POS tagset and the larger set of lexical types from Combinatory Categorial Grammar. 1 Introduction The use of maximum entropy (ME) models has become popular in Statistical NLP; some exam- ple applications include part-of-speech (Pos) tag- ging (Ratnaparkhi, 1996), parsing (Ratnaparkhi, 1999; Johnson et al., 1999) and language mod- elling (Rosenfeld, 1996). Many tagging problems have been successfully modelled in the ME frame- work, including POS tagging, with state of the art performance (van Halteren et al., 2001), "su- pertagging" (Clark, 2002) and chunking (Koeling, 2000). Generalised Iterative Scaling (GIs) is a very simple algorithm for estimating the parameters of a ME model. The original formulation of GIS (Dar- roch and Ratcliff, 1972) required the sum of the feature values for each event to be constant. Since this is not the case for many applications, the stan- dard method is to add a "correction", or "slack", feature to each event Improved Iterative Scal- ing (us) (Berger et al., 1996; Della Pietra et al., 1997) eliminated the correction feature to improve the convergence rate of the algorithm. However, the extra book keeping required for us means that GIS is often faster in practice (Malouf, 2002). This paper shows, by a simple adaptation of Berger's proof for the convergence of HS (Berger, 1997), that GIS does not require a correction feature. We also investigate how the use of a correction feature affects the performance of ME taggers. GIS and HS obtain a maximum likelihood es- timate (mLE) of the parameters, and, like other MLE methods, are susceptible to overfitting. A simple technique used to avoid overfitting is a fre- quency cutoff, in which only frequently occurring features are included in the model (Ratnaparkhi, 1998). However, more sophisticated smoothing techniques exist, such as the use of a Gaussian prior on the parameters of the model (Chen and Rosenfeld, 1999). This technique has been ap- plied to language modelling (Chen and Rosenfeld, 1999), text classification (Nigam et al., 1999) and parsing (Johnson et al., 1999), but to our knowl- edge it has not been compared with the use of a feature cutoff. We explore the combination of Gaussian smoothing and a simple cutoff for two tagging tasks. The two taggers used for the experiments are a POS tagger, trained on the WSJ Penn Treebank, and a "supertagger", which assigns tags from the 91 much larger set of lexical …

Content truncated to 500 words