On the Role of Lexical Features in Sequence Labeling

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1142–1151, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP On the Role of Lexical Features in Sequence Labeling Yoav Goldberg∗and Michael Elhadad Ben Gurion University of the Negev Department of Computer Science POB 653 Be’er Sheva, 84105, Israel {yoavg|elhadad}@cs.bgu.ac.il Abstract We use the technique of SVM anchoring to demonstrate that lexical features extracted from a training corpus are not necessary to obtain state of the art results on tasks such as Named Entity Recognition and Chunk- ing. While standard models require as many as 100K distinct features, we derive models with as little as 1K features that perform as well or better on different do- mains. These robust reduced models in- dicate that the way rare lexical features contribute to classification in NLP is not fully understood. Contrastive error analy- sis (with and without lexical features) in- dicates that lexical features do contribute to resolving some semantic and complex syntactic ambiguities – but we find this contribution does not generalize outside the training corpus. As a general strat- egy, we believe lexical features should not be directly derived from a training corpus but instead, carefully inferred and selected from other sources. 1 Introduction Common NLP tasks, such as Named Entity Recognition and Chunking, involve the identifi- cation of spans of words belonging to the same phrase. These tasks are traditionally reduced to a tagging task, in which each word is to be clas- sified as either Beginning a span, Inside a span, or Outside of a span. The decision is based on the word to be classified and its neighbors. Fea- tures supporting the classification usually include ∗Supported by the Lynn and William Frankel Center for Computer Sciences, Ben Gurion University the word forms themselves and properties derived from the word forms, such as prefixes, suffixes, capitalization information, and parts-of-speech. While early approaches to the NP-chunking task (Cardie and Pierce, 1998) relied on part-of-speech information alone, it is widely accepted that lexi- cal information (word forms) is crucial for build- ing accurate systems for these tasks. Indeed, all the better-performing systems in the CoNLL shared tasks competitions for Chunking (Sang and Buchholz, 2000) and Named Entity Recognition (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003) make extensive use of such lexical information. Is this belief justified? In this paper, we show that the influence of lexical features on such se- quence labeling tasks is more complex than is gen- erally assumed. We find that exact word forms aren’t necessary for accurate classification. This observation is important because relying on the exact word forms that appear in a training corpus leads to over-fitting, as well as to larger models. In this work, we focus on learning with Support Vector Machines (SVMs) (Vapnik, 1995). SVM classifiers can handle very large feature spaces, and produce state-of-the-art results for NLP ap- plications (see e.g. (Kudo and Matsumoto, 2000; Nivre et al., 2006)). Alas, when trained on pruned feature …

Content truncated to 500 words