Crowdsourcing the evaluation of a domain-adapted named entity recognition system

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 345–348, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Crowdsourcing the evaluation of a domain-adapted named entity recognition system Asad B. Sayeed, Timothy J. Meyer, Hieu C. Nguyen, Olivia Buzek Department of Computer Science University of Maryland College Park, MD 20742 asayeed@cs.umd.edu, tmeyer1@umd.edu, {hcnguyen88,olivia.buzek} @gmail.com Amy Weinberg Department of Linguistics University of Maryland College Park, MD 20742 weinberg@umiacs.umd.edu Abstract Named entity recognition systems sometimes have difficulty when applied to data from do- mains that do not closely match the training data. We first use a simple rule-based tech- nique for domain adaptation. Data for robust validation of the technique is then generated, and we use crowdsourcing techniques to show that this strategy produces reliable results even on data not seen by the rule designers. We show that it is possible to extract large im- provements on the target data rapidly at low cost using these techniques. 1 Introduction 1.1 Named entities and errors In this work, we use crowdsourcing to generate eval- uation data to validate simple techniques designed to adapt a widely-used high-performing named entity recognition system to new domains. Specifically, we achieve a roughly 10% improvement in precision on text from the information technology (IT) business press via post hoc rule-based error reduction. We first tested the system on a small set of data that we annotated ourselves. Then we collected data from Amazon Mechanical Turk in order to demonstrate that the gain is stable. To our knowledge, there is no previous work on crowdsourcing as a rapid means of evaluating error mitigation in named entity rec- ognizer development. Named entity recognition (NER) is a well-known problem in NLP which feeds into many other re- lated tasks such as information retrieval (IR) and machine translation (MT) and more recently social network discovery and opinion mining. Generally, errors in the underlying NER technology correlate with a steep price in performance in the NLP sys- tems further along a processing pipeline, as incor- rect entities propagate into incorrect translations or erroneous graphs of social networks. Not all errors carry the same price. In some ap- plications, omitting a named entity has the conse- quence of reducing the availability of training data, but including an incorrectly identified piece of text as as a named entity has the consequence of pro- ducing misleading results. Our application would be opinion mining; an omitted entity may prevent the system from attributing an opinion to a source, but an incorrect entity reveals non-existent opinion sources. Machine learning is currently used extensively in building NER systems. One such system is BBN’s Identifinder (Bikel et al., 1999). The IdentiFinder al- gorithm, based on Hidden Markov Models, has been shown to achieve F-measure scores above 90% when the training and testing data happen to be derived from Wall Street Journal text produced in the 1990s. We use IdentiFinder 3.3 as a starting point for per- formance improvement in this paper. …

Content truncated to 500 words