Survey of the Message Understanding Conferences

SURVEY OF THE MESSAGE UNDERSTANDING CONFERENCES Beth M. Sundheim Naval Command, Control & Ocean Surveillance Ctr. RDT&E Division (NRaD), Code 444 San Diego, CA 92152-7420 Nancy A. Chinchor Science Applications International Corporation 10260 Campus Point Drive, M/S A2-F San Diego, CA 92121 ABSTRACT In this paper, the Message Understanding Conferences are reviewed, and the natural language system evaluation that is underway in preparation for the next conference is described. The role of the conferences in the evaluation of information extraction systems is assessed in terms of the purposes of three broad classes of evaluation: progress, adequacy, and diagnostic. The conferences have measured system performance primarily to assess progress and the state of the art, but they have also been influenced by the concerns associated with assessing adequacy and providing diagnostics. Challenges for the future of similar evaluations are also discussed. 1. INTRODUCTION Much has happened since the last time a paper appeared in the ARPA workshop proceedings about the Message Understanding Conferences [11]. The evaluation methodology has been changing steadily, and more demanding information extraction tasks have been defined. In response to the challenges of the evaluation task and metrics, researchers have developed robust and efficient methods for working with large corpora and have confronted prevalent text analysis issues that have so far conswained performance. These challenges have also resulted in a critical rethinking of assumptions concerning the ideal system to submit for evaluation. Is it a "generic" natural language system with in-depth analysis capabilities and a well- defined internal representation language designed to accomodate the translation of various kinds of textual input into various kinds of output? Or is it one that uses only shallow processing techniques and does not presume to be suitable for language processing tasks other than information extraction? 2. REVIEW OF PAST MUCs The first Message Understanding Conference (MUC) was held in 1987, used ten narrative paragraphs from naval messages as a training corpus and two others as test data, and had no defined evaluation task or metrics. Researchers from six organizations ran their systems on the test data during the conference, then demonstrated and explained how the systems analyzed the texts. Two years later, the second MUC was held [10]. It made use of a training corpus of 105 naval message narratives of four different types, a dry-run test set of 20 narratives, and a final test set of five. An information extraction task was defined that consisted of identifying ten different pieces of information and representing them as slot fillers in a template resembling a semantic frame. This task emulates an information management application requiring the culling of facts from a large body of free text as a means to generate updates to a formatted database. A rudimentary set of scoring standards was developed, and the templates produced by the eight systems (including four of the six systems represented at the 1987 evaluation) were scored by hand by comparison with a hand-generated answer key. The nature of the corpus used for the second …

Content truncated to 500 words