Shallow Parsing with Conditional Random Fields

Shallow Parsing with Conditional Random Fields Fei Sha and Fernando Pereira Department of Computer and Information Science University of Pennsylvania 200 South 33rd Street, Philadelphia, PA 19104 (feisha|pereira)@cis.upenn.edu Abstract Conditional random fields for sequence label- ing offer advantages over both generative mod- els like HMMs and classifiers applied at each sequence position. Among sequence labeling tasks in language processing, shallow parsing has received much attention, with the devel- opment of standard evaluation datasets and ex- tensive comparison among methods. We show here how to train a conditional random field to achieve performance as good as any reported base noun-phrase chunking method on the CoNLL task, and better than any reported sin- gle model. Improved training methods based on modern optimization algorithms were crit- ical in achieving these results. We present ex- tensive comparisons between models and train- ing methods that confirm and strengthen pre- vious results on shallow parsing and training methods for maximum-entropy models. 1 Introduction Sequence analysis tasks in language and biology are of- ten described as mappings from input sequences to se- quences of labels encoding the analysis. In language pro- cessing, examples of such tasks include part-of-speech tagging, named-entity recognition, and the task we shall focus on here, shallow parsing. Shallow parsing iden- tifies the non-recursive cores of various phrase types in text, possibly as a precursor to full parsing or informa- tion extraction (Abney, 1991). The paradigmatic shallow- parsing problem is NP chunking, which finds the non- recursive cores of noun phrases called base NPs. The pioneering work of Ramshaw and Marcus (1995) in- troduced NP chunking as a machine-learning problem, with standard datasets and evaluation metrics. The task was extended to additional phrase types for the CoNLL- 2000 shared task (Tjong Kim Sang and Buchholz, 2000), which is now the standard evaluation task for shallow parsing. Most previous work used two main machine-learning approaches to sequence labeling. The first approach re- lies on k-order generative probabilistic models of paired input sequences and label sequences, for instance hidden Markov models (HMMs) (Freitag and McCallum, 2000; Kupiec, 1992) or multilevel Markov models (Bikel et al., 1999). The second approach views the sequence labeling problem as a sequence of classification problems, one for each of the labels in the sequence. The classification re- sult at each position may depend on the whole input and on the previous k classifications. 1 The generative approach provides well-understood training and decoding algorithms for HMMs and more general graphical models. However, effective genera- tive models require stringent conditional independence assumptions. For instance, it is not practical to make the label at a given position depend on a window on the in- put sequence as well as the surrounding labels, since the inference problem for the corresponding graphical model would be intractable. Non-independent features of the inputs, such as capitalization, suffixes, and surrounding words, are important in dealing with words unseen in training, but they are difficult to represent in generative models. The sequential classification approach can handle many correlated features, as …

Content truncated to 500 words