A Methodology for Extrinsically Evaluating Information Extraction Performance

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 652–659, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics A Methodology for Extrinsically Evaluating Information Extraction Performance Michael Crystal, Alex Baron, Katherine Godfrey, Linnea Micciulla, Yvette Tenney, and Ralph Weischedel BBN Technologies 10 Moulton St. Cambridge, MA 02138-1119 mcrystal@bbn.com Abstract This paper reports a preliminary study addressing two challenges in measuring the effectiveness of information extrac- • tion (IE) technology: Developing a methodology for ex- trinsic evaluation of IE; and, • Estimating the impact of improving IE technology on the ability to per- form an application task. The methodology described can be em- ployed for further controlled experi- ments regarding information extraction. 1 Introduction Intrinsic evaluations of information extraction (IE) have a history dating back to the Third Mes- sage Understanding Conference1 (MUC-3) and continuing today in the Automatic Content Ex- traction (ACE) evaluations.2 Extrinsic evalua- tions of IE, measuring the utility of IE in a task, are lacking and needed (Jones, 2005). In this paper, we investigate an extrinsic evaluation of IE where the task is question an- swering (QA) given extracted information. In addition, we propose a novel method for explor- ing hypothetical performance questions, e.g., if IE accuracy were x% closer to human accuracy, how would speed and accuracy in a task, e.g., QA, improve? 1 For more information on the MUC conferences, see http://www.itl.nist.gov/iad/894.02/related_projects/muc/. 2 For an overview of ACE evaluations see http://www.itl.nist.gov/iad/894.01/tests/ace/. We plot QA accuracy and time-to-complete given eight extracted data accuracy levels rang- ing from the output of SERIF, BBN’s state-of- the-art IE system, to manually extracted data. 2 Methodology Figure 1 gives an overview of the methodol- ogy. The left portion of the figure shows source documents provided both to a system and a hu- man to produce two extraction databases, one corresponding to SERIF’s automated perform- ance and one corresponding to double- annotated, human accuracy. By merging por- tions of those two sources in varying degrees (“blends”), one can derive several extracted da- tabases ranging from machine quality, through varying percentages of improved performance, up to human accuracy. This method of blending databases provides a means of answering hypo- thetical questions, i.e., what if the state-of-the- art were x% closer to human accuracy, with a single set of answer keys. A person using a given extraction database performs a task, in our case, QA. The measures of effectiveness in our study were time to com- plete the task and percent of questions answered correctly. An extrinsic measure of the value of improved IE technology performance is realized by rotating users through different extraction databases and questions sets. In our preliminary study, databases of fully automated IE and manual annotation (the gold standard) were populated with entities, relation- ships, and co-reference links from 946 docu- ments. The two initial databases representing machine extraction and human extraction re- spectively were then blended to produce a con- tinuum of database qualities from machine to 652 human performance. …

Content truncated to 500 words