Named Entity Recognition using Hundreds of Thousands of Features

Named Entity Recognition using Hundreds of Thousands of Features James Mayfield and Paul McNamee and Christine Piatko The Johns Hopkins University Applied Physics Laboratory 11100 Johns Hopkins Road, Laurel, Maryland 20723-6099 USA {mayfield,mcnamee,piatko}@jhuapl.edu Abstract We present an approach to named entity recog- nition that uses support vector machines to cap- ture transition probabilities in a lattice. The support vector machines are trained with hun- dreds of thousands of features drawn from the CoNLL-2003 Shared Task training data. Mar- gin outputs are converted to estimated prob- abilities using a simple static function. Per- formance is evaluated using the CoNLL-2003 Shared Task test set; Test B results were Fβ=1 = 84.67 for English, and Fβ=1 = 69.96 for Ger- man. 1 Introduction Language independence is difficult to achieve in named entity recognition (NER) because different languages ap- pear to require different features. Most NER systems (or taggers) are severely limited in the number of features they may consider, because the computational expense of handling large numbers of features is high, and because the risk of overtraining increases with the number of fea- tures. Thus, the feature set must be finely tuned to be effective. Such constrained feature sets are naturally lan- guage dependent. Increasing the number of features that a tagger can han- dle would ameliorate this problem, because the designer could select many relatively simple features in lieu of a few highly tuned features. Because support vector ma- chines (SVMs) (Vapnik, 1995) can handle large numbers of parameters efficiently while simultaneously limiting overtraining, they are good candidates for application to named entity recognition. This paper proposes a novel way to use SVMs for named entity recognition called SVM-Lattice, describes a large feature space that we used on the CoNLL-2003 Shared Task (Tjong Kim Sang and De Meulder, 2003), and presents results from that task. 2 Model We are interested in a lattice-based approach to named entity recognition. In this approach, each sentence is pro- cessed individually. A lattice is built with one column per word of the sentence (plus a start state). Each column contains one vertex for each possible tag. Each vertex in one column is connected by an edge to every vertex in the next column that may legitimately follow it (some tran- sitions, such as from I-LOC to B-PER are disallowed). Given such a lattice, our task is first to assign probabili- ties to each of the arcs, then to find the highest likelihood path through the lattice based on those probabilities. This path corresponds to the highest likelihood tagging of the sentence. Hidden Markov models break the probability calcula- tions into two pieces: transition probabilities (the proba- bility of moving from one vertex to another independent of the word at the destination node), and emission proba- bilities (the probability that a given word would be gener- ated from a certain state independent of the path taken to get to that state). These probability distributions are cal- culated separately because the training data are typically too sparse to …

Content truncated to 500 words