Analysing Wikipedia and Gold-Standard Corpora for NER Training

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 612–620, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Analysing Wikipedia and Gold-Standard Corpora for NER Training Joel Nothman and Tara Murphy and James R. Curran School of Information Technologies University of Sydney NSW 2006, Australia {jnot4610,tm,james}@it.usyd.edu.au Abstract Named entity recognition (NER) for En- glish typically involves one of three gold standards: MUC, CoNLL, or BBN, all created by costly manual annotation. Recent work has used Wikipedia to automatically cre- ate a massive corpus of named entity an- notated text. We present the first comprehensive cross- corpus evaluation of NER. We identify the causes of poor cross-corpus perfor- mance and demonstrate ways of making them more compatible. Using our process, we develop a Wikipedia corpus which out- performs gold standard corpora on cross- corpus evaluation by up to 11%. 1 Introduction Named Entity Recognition (NER), the task of iden- tifying and classifying the names of people, organ- isations and other entities within text, is central to many NLP systems. NER developed from in- formation extraction in the Message Understand- ing Conferences (MUC) of the 1990s. By MUC 6 and 7, NER had become a distinct task: tagging proper names, and temporal and numerical expres- sions (Chinchor, 1998). Statistical machine learning systems have proven successful for NER. These learn patterns associated with individual entity classes, mak- ing use of many contextual, orthographic, linguis- tic and external knowledge features. However, they rely heavily on large annotated training cor- pora. This need for costly expert annotation hin- ders the creation of more task-adaptable, high- performance named entity recognisers. In acquiring new sources for annotated corpora, we require an analysis of training data as a variable in NER. This paper compares the three main gold- standard corpora. We found that tagging mod- els built on each corpus perform relatively poorly when tested on the others. We therefore present three methods for analysing internal and inter- corpus inconsistencies. Our analysis demonstrates that seemingly minor variations in the text itself, starting right from tokenisation can have a huge impact on practical NER performance. We take this experience and apply it to a corpus created automatically using Wikipedia. This cor- pus was created following the method of Nothman et al. (2008). By training the C&C tagger (Curran and Clark, 2003) on the gold-standard corpora and our new Wikipedia-derived training data, we eval- uate the usefulness of the latter and explore the nature of the training corpus as a variable in NER. Our Wikipedia-derived corpora exceed the per- formance of non-corresponding training and test sets by up to 11% F-score, and can be engineered to automatically produce models consistent with various NE-annotation schema. We show that it is possible to automatically create large, free, named entity-annotated corpora for general or domain specific tasks. 2 NER and annotated corpora Research into NER has rarely considered the im- pact of training corpora. The CoNLL evalua- tions focused on machine learning methods (Tjong …

Content truncated to 500 words