Feature Subset Selection in Conditional Random Fields for Named Entity Recognition

International Conference RANLP 2009 - Borovets, Bulgaria, pages 185–191 Feature Subset Selection in Conditional Random Fields for Named Entity Recognition Roman Klinger and Christoph M. Friedrich Department of Bioinformatics Fraunhofer Institute for Algorithms and Scientific Computing (SCAI) 53754 Sankt Augustin, Germany {roman.klinger,christoph.friedrich}@scai.fraunhofer.de Abstract In the application of Conditional Random Fields (CRF), a huge number of features is typically taken into ac- count. These models can deal with inter-dependent and correlated data with an enormous complexity. The application of feature subset selection is important to improve performance, speed and explainability. We present and compare filtering methods using infor- mation gain or χ2 as well as an iterative approach for pruning features with low weights. The evaluation shows that with only 3 % of the orig- inal number of features a 60 % inference speed-up is possible. The F1 measure decreases only slightly. 1 Introduction Feature selection is well established for many machine learn- ing methods, for instance for feed-forward neural networks [2] or decision trees [17]. The main advantages are an im- provement of prediction performance, faster training and prediction as well as a better understanding of the models [4]. Methods can be distinguished between filters not us- ing the learning algorithm and wrappers using the learning algorithm as a black box [8]. An overview of approaches for classification tasks is given by Liu and Motoda [11], more specifically for text classification by Yang and Peder- son [25]. Such feature selection methods are not well established for Conditional Random Fields [9], a Maximum Entropy- based method [1] for structured data. We propose methods coping with the demanding task of handling sequential data represented by a huge number of features. Reported num- bers are for instance 1,686,456 for a gene name tagger [5]. Due to this high complexity, training and inference times can explode. These high numbers of features are generated by automated methods, i. e., for every token in the train- ing set, features are generated. Then, all other tokens are tested for these features. This method is typically applied to determine the identity of words as well as for prefixes or suffixes of different length or for learning schemata of regular expression-like patterns [20]. Only a few approaches dealing with feature handling for CRFs are published. The work by McCallum [12] demon- strates a method for iteratively constructing feature con- junctions that would increase conditional log-likelihood if added to the model. An analysis of different penalty terms for regularization is shown by Peng and McCallum [15]. Goodman [3] presented a related analysis of exponential priors for Maximum Entropy models. The very recent work of Vail et al. [23, 22] shows feature selection in Conditional Random Fields by L1-norm regularization in the robotics domain, a work which is related to the selection in Maxi- mum Entropy models proposed by Koh et al. [7]. These methods incorporate the training procedure in the selection process. In contrast, we present different filter methods for feature selection to limit the complexity before starting the training. …

Content truncated to 500 words