GE Adjunct Test Report: Object-Oriented Design and Scoring for MUC-4

GE ADJUNCT TEST REPORT : OBJECT-ORIENTED DESIGN AND SCORING FOR MUC- 4 George Krupka and Lisa Rau Artificial Intelligence Laborator y GE Research and Development Schenectady, NY 12301 USA E-mail: rau@crd.ge.com Phone: (518) 387 - 5059 Abstract This paper reports on the results of the adjunct test performed by GE for the MUC-4 evaluation of text processing systems . In this test, we evaluated the effect of an object-oriented templat e design and associated matching conditions on the scores . The results indicate that the current MUC- .{ "flat" templade design with cross-references closely approximates a true object-oriente d design. However the object-oriented design allows for additional performance data to be calculated , facilitating diagnosis . INTRODUCTION In this adjunct test, we investigate the issues and effect of transforming the MUC-4 template design au- tomatically into an object-oriented design, with associated object-level matching conditions affecting th e overall score . An object is simply a collection of slots that all refer to one item originating in the text. This collection of slots is logically connected due to their implicit reference to one particular filler . In addition, objects may be nested so that one object contains another object, or may be recursive, with one object pointing t o another and back again . Finally, there may be multiple instances of any given object . In MUC-4 is is possible to isolate three distinct levels of objects . The first level contains a STORY object . Attached to a STORY object are INCIDENT objects, each of which contains participant objects ; TARGET , INSTRUMENT and PERPETRATOR. The Figure 1 illustrates an object-oriented template design . The MUC template design has been moving from a flat structure to a more object-oriented structur e with cross-references tying together multiple slots into what has been termed a "pseudo-object" . Slots tie d together with cross-references are pseudo-objects and not true objects because the cross-references are not enforced. For example, consider the following scenario depicted in Figure 2 . Although the human target ("MARY") is wrong, the system is allowed partial credit for the other slots, even though they are clearl y cross-referenced to different, and equal incorrect, fills . MOTIVATIO N There are a variety of reasons why an object-oriented design is desirable. First, an object-oriented design is conceptually easier to understand. Instead of a flat listing of all slots of a template, slots pertaining to a single fill are grouped together. Cross-references are no longer needed, as indentation indicates the grouping , so the visual design is cleaner . Asthetics aside, there are additional performance data that can be obtained when groups of slots ar e connected as objects . Systems can be scored on how well a given object aligns, in a way analogous t o template matching alignment scores (template ID score) . Moreover, it is possible to construct an object total that does not consider whether any given object appears in a matching template or not . For exampl e …

Content truncated to 500 words