TIPSTER SHOGUN System (Joint GE-CMU): MUC-4 Test Results and Analysis

TIPSTER SHOGUN SYSTEM (JOINT GE-CMU) : MUC-4 TEST RESULTS AND ANALYSIS 1 George Krupka and Paul Jacob s Artificial Intelligence Laboratory GE Research and Development Schenectady, NY 12301 US A E-mail: rau@crd .ge.com Phone: (518) 387 - 5059 Michael Mauldin and Todd Kaufman n Carnegie Mellon Universit y Ira Sider Military and Data Systems Operatio n GE Aerospace Abstract This paper reports on the joint GE-CMU Tipster SHOGUN customization effort for MUC-4 , and analyzes the results of the TST3 and TST4 runs in comparison with the GE system . INTRODUCTION We report on the joint GE-CMU results from the MUC-4 conference and provide an analysis of syste m performance. In its maiden test, the joint SHOGUN system performed very well . On the positive side, the system achieved very good overall results. On the negative side, because the system was barely ready in time for MUC-3, it was difficult to implement and test any significant modifications . RESULTS Our overall results on both TST3 and TST4 were very good in relation to other systems . Figure 1 summarizes our results on these tests . On both TST3 and TST4, the GE-CMU system was 9-10 recall points behin d the GE system and about 4 F-measure points behind . The entire difference is due to the difference between the GE TRUMP parser, which had been developed for text applications and thoroughly tested in MUC-3 , and the CMU generalized LR parser, which was developed for machine translation and has just begun to b e tested on this sort of task . In addition to these core results, the Figure 2 summarizes our performance on the adjunct test . GE - GE-CMU SCORE COMPARISO N While the performance of the GE-CMU system was not as good as that of the GE system, we view thes e results in a very positive light. First, the system with the CMU parser was reasonably close to the G E system, and it was rapidly catching up at the end of the preparations for MUC-4 (The GE-CMU syste m improved about 30 recall points between the interim test in March and the final test in May) . Second , the integrated system proved that software modules and fundamental results could be shared across sites , This research was sponsored (in part) by the Defense Advanced Research Project Agency (DOD) and other governmen t agencies. The views and conclusions contained in this document are those of the authors and should not be interpreted a s representing the official policies, either expressed or implied, of the Defense Advanced Research Project Agency or the U S Government . 100 TST3 TST4 REC PRE OVG F Matched Only Templates GE AT All 66 72 13 69 68 1 8 49 55 33 51.8 53 53 37 53 58 54 36 55.9 62 53 39 57.15 Figure 1 : GE-CMU MUC-4 TST3 and TST4 Result s 1MT 1ST 2MT NST TST-3 60 58 29 58.9 57 35 63 43.4 34 …

Content truncated to 500 words