Named Entity Recognition with Character-Level Models

Named Entity Recognition with Character-Level Models Dan Klein � and Joseph Smarr � and Huy Nguyen � and Christopher D. Manning � � Computer Science Dept. � Symbolic Systems Program Stanford University Stanford University Stanford, CA 94305-9040 Stanford, CA 94305-2181 � klein,htnguyen,manning � @cs.stanford.edu jsmarr@stanford.edu Abstract We discuss two named-entity recognition mod- els which use characters and character � -grams either exclusively or as an important part of their data representation. The first model is a character-level HMM with minimal con- text information, and the second model is a maximum-entropy conditional markov model with substantially richer context features. Our best model achieves an overall F � of 86.07% on the English test data (92.31% on the devel- opment data). This number represents a 25% error reduction over the same model without word-internal (substring) features. 1 Introduction For most sequence-modeling tasks with word-level eval- uation, including named-entity recognition and part-of- speech tagging, it has seemed natural to use entire words as the basic input features. For example, the classic HMM view of these two tasks is one in which the ob- servations are words and the hidden states encode class labels. However, because of data sparsity, sophisti- cated unknown word models are generally required for good performance. A common approach is to extract word-internal features from unknown words, for example suffix, capitalization, or punctuation features (Mikheev, 1997, Wacholder et al., 1997, Bikel et al., 1997). One then treats the unknown word as a collection of such fea- tures. Having such unknown-word models as an add-on is perhaps a misplaced focus: in these tasks, providing correct behavior on unknown words is typically the key challenge. Here, we examine the utility of taking character se- quences as a primary representation. We present two models in which the basic units are characters and char- acter � -grams, instead of words and word phrases. Ear- lier papers have taken a character-level approach to named entity recognition (NER), notably Cucerzan and Yarowsky (1999), which used prefix and suffix tries, though to our knowledge incorporating all character � - grams is new. In section 2, we discuss a character-level HMM, while in section 3 we discuss a sequence-free maximum-entropy(maxent) classifier which uses � -gram substring features. Finally, in section 4 we add additional features to the maxent model, and chain these models into a conditional markov model (CMM), as used for tag- ging (Ratnaparkhi, 1996) or earlier NER work (Borth- wick, 1999). 2 A Character-Level HMM Figure 1 shows a graphical model representation of our character-level HMM. Characters are emitted one at a time, and there is one state per character. Each state’s identity depends only on the previous state. Each char- acter’s identity depends on both the current state and on the previous ��� characters. In addition to this HMM view, it may also be convenient to think of the local emis- sion models as type-conditional � -gram models. Indeed, the character emission model in this section is directly based on the � -gram proper-name classification …

Content truncated to 500 words