GE NLToolset: MUC-3 Test Results and Analysis

GE NLTOOLSET : MUC-3 TEST RESULTS AND ANALYSI S George Krupka, Lucja Iwariska, Paul Jacobs and Lisa Ra u Artificial Intelligence Laboratory GE Research and Developmen t Schenectady, NY 12301 US A ra.u@crd .ge.com Abstract This paper reports on the GE NLTooLsET customization effort for MUC-3, and analyzes th e results of the TST2 run . Although our own tests had shown steady improvement between TST1 and TST2, our official scores on TST2 were lower than on TST1 . The analysis of this unexpected result explains some of the details of the i1'UC-3 test, and we propose ways of looking at the scores . to distinguish different aspects of system performance . INTRODUCTION We report on the GE results from the MUC-3 conference and provide an analysis of system performance . In general, MUC-3 was a very successful effort for GE. The NLTooLsET, a suite of natural language tex t processing tools designed for easy application in new domains, proved its mettle, as we were quickly able t o show good results on the MUC-3 task . Even on TST2, where we experienced some system-level problems , all of our results were in the top group, and the program was especially accurate at filling out templates There were, however, some surprises that resulted from MUC-3, including the major differences in syste m. capabilities that are largely hidden in the scores, as well as the relative ease of sentence-level interpretation . On the positive side, MUC-3 provided a thorough, fair test of system capabilities . The methodology of testing on a real task, along with the benefit of a common corpus, has produced advances in the fiel d as well as highlighting certain new aspects of text interpretation . Certain parts of our system, includin g a lexically-driven pre-processor and knowledge-based language analysis mechanism, worked extremely well , while other issues, such as our lack of an explicit discourse representation, prevented us from doing better . This recognition of strengths and weaknesses comes directly from the results of the MUC-3 experiment . On the negative side, too much of each system's score, as well as the work involved in the task, is fro m applying "rules of the game", and future MUCs must try to minimize this component . Some of these rule s are not tied either to text processing capabilities or to the practical requirements of the task . Another problem is how to determine from the results what a system is actually doing, as some major difference s between systems proved largely hidden in the MUC-3 scoring . In this report, we will attempt to relate th e evaluation results to system capabilities as well as to suggest some methods for attributing different aspect s of the scores to particular types of processing . RESULTS Our overall results on TST2 were very good in relation to other systems, but we devoted much of our analysi s to explaining why they were much lower than our expectations . The …

Content truncated to 500 words