Comparing Human and Machine Performance for Natural Language Information Extraction: Results from the Tipster Text Evaluation

COMPARING HUMAN AND MACHINE PERFORMANCE FOR NATURAL LANGUAGE INFORMATION EXTRACTION: Results from the Tipster Text Evaluation Craig A. Will Institute for Defense Analyses Computer and Software Engineering Division 1801 N. Beauregard Street Alexandria, VA 22311 ABSTRACT This paper presents results from a study comparing human perfor- mance on the text of natural language information extraction with that of machine extraction systems that were developed as part of the ARPA Tipster program. Information extraction is shown to be a difficult task for both humans and machines. Evidence for one set of text material, English Microelectronics, indicated that a human analyst produces about half the errors as does machine systems. INTRODUCTION In evaluating the state of technology for extracting informa- tion from natural language text by machine, it is valuable to compare the performance of machine extraction systems with that achieved by humans performing the same task. The purpose of this paper is to present some results from a comparative study of human and machine performance for one of the information extraction tasks used in the Tipster/ MUC-5 evaluation that can help assess the maturity and applicability of the technology. The Tipster program, through the Institute for Defense Analyses (IDA) and several collaborating U.S. government agencies, produced a corpus of filled "templates" --StlUC- tured information extracted from text. This corpus was used both in the development of machine extraction systems by contractors and in the evaluation of the developed systems. Production of templates was performed by human analysts extracting the data from the text and structuring it, using a set of structuring rules for "filling" the templates and com- puter software that made it easier for analysts to organize information. Because of this rather extensive effort by ana- lysts to create these templates, it was possible to study the performance of humans for this task in some detail and to develop methods for comparing this performance with that of machines participating in the Tipster/MUC-5 evaluation. The texts that the templates were filled from were newspa- per and technical magazine articles concerned either with joint business ventures or microelectronics fabrication tech- nology. Each topic domain used text in two languages, English and Japanese. This paper discusses preparation of templates and presents detailed results for human and machine performance; a shorter paper [1] discusses prepa- ration of templates and basic results. The primary motivation for this study was to provide reli- able data that would allow machine extraction performance to be compared with that of humans. The MUC and Tipster programs have included extensive efforts to develop mea- surements that can objectively evaluate the performance of the different machine systems. However, although these measures are capable of reliably discriminating between the performance of different machine systems, they are not very useful by themselves in evaluating how near the technology is to providing reliable performance and the extent to which it is ready to be used in applications. Sundheim [2] initiated human performance study for extraction by providing esti- mates of human performance for the …

Content truncated to 500 words