Empirical Study on the Performance Stability of Named Entity Recognition Model across Domains

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 509–516, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Empirical Study on the Performance Stability of Named Entity Recognition Model across Domains Hong Lei Guo Li Zhang and IBM China Research LaboratoryZhong Su Building 19, Zhongguancun Software Park 8 Dongbeiwang WestRoad, Haidian District, Beijing, 100094, P.R.C. {guohl, lizhang , suzhong }@cn.ibm.com Abstract When a machine learning-based named entity recognition system is employed in a new domain, its performance usually de- grades. In this paper, we provide an em- pirical study on the impact of training data size and domain information on the per- formance stability of named entity recog- nition models. We present an informative sample selection method for building high quality and stable named entity recogni- tion models across domains. Experimen- tal results show that the performance of the named entity recognition model is en- hanced significantly after being trained with these informative samples. 1 Introduction Named entities (NE) are phrases that contain names of persons, organizations, locations, etc. Named entity recognition (NER) is an important task in many natural language processing appli- cations, such as information extraction and ma- chine translation. There have been a number of conferences aimed at evaluating NER systems, for example, MUC6, MUC7, CoNLL2002 and CoNLL2003, and ACE (automatic content extrac- tion) evaluations. Machine learning approaches are becoming more attractive for NER in recent years since they are trainable and adaptable. Recent research on English NER has focused on the machine learning approach (Sang and Meulder, 2003). The relevant algorithms include Maximum Entropy (Borth- wick, 1999; Klein et al., 2003), Hidden Markov Model (HMM) (Bikel et al., 1999; Klein et al., 2003), AdaBoost (Carreras et al., 2003), Memory- based learning (Meulder and Daelemans, 2003), Support Vector Machine (Isozaki and Kazawa, 2002), Robust Risk Minimization (RRM) Classi- fication method (Florian et al., 2003), etc. For Chinese NER, most of the existing ap- proaches use hand-crafted rules with word (or character) frequency statistics. Some machine learning algorithms also have been investigated in Chinese NER, including HMM (Yu et al., 1998; Jing et al., 2003), class-based language model (Gao et al., 2005; Wu et al., 2005), RRM (Guo et al., 2005; Jing et al., 2003), etc. However, when a machine learning-based NER system is directly employed in a new domain, its performance usually degrades. In order to avoid the performance degrading, the NER model is of- ten retrained with domain-specific annotated cor- pus. This retraining process usually needs more efforts and costs. In order to enhance the perfor- mance stability of NER models with less efforts, some issues have to be considered in practice. For example, how much training data is enough for building a stable and applicable NER model? How does the domain information and training data size impact the NER performance? This paper provides an empirical study on the impact of training data size and domain informa- tion on NER performance. Some useful observa- tions are obtained from the experimental …

Content truncated to 500 words