Algorithms That Learn to Extract Information BBN: TIPSTER Phase III

ALGORITHMS THAT LEARN TO EXTRACT INFORMATION m BBN: TIPSTER PHASE III Scott Miller, Michael Crystal, Heidi Fox, Lance Ramshaw, Richard Schwartz, Rebecca Stone, and Ralph Weischedel BBN Technologies 70 Fawcett Street Cambridge, MA 02138 weischedel @bbn.com ABSTRACT All of BBN's research under the TIPSTER III program has focused on doing extraction by applying statistical models trained on annotated data, rather than by using programs that execute hand-written rules. Within the context of MUC- 7, the SIFT system for extraction of template entities (TE) and template relations (TR) used a novel, integrated syntactic/semantic language model to extract sentence level information, and then synthesized information across sentences using in part a trained model for cross-sentence relations. At the named entity (NE) level as well, in both MET-1 and MUC-7, BBN employed a trained, HMM-based model. The results in these TIPSTER evaluations are evidence that such trained systems, even at their current level of development, can perform roughly on a par with those based on rules hand- tailored by experts. In addition, such trained systems have some significant advantages: • They can be easily ported to new domains by simply annotating fresh data. • The complex interactions that make rule- based systems difficult to develop and maintain can here be learned automatically from the training data. We believe that improved and extended versions of such trained models have the potential for significant further progress toward practical systems for information extraction. INTRODUCTION We believe that trained statistical models offer significant advantages for information extraction tasks. In this report on BBN's research under the TIPSTER III program, we describe a number of research efforts that developed fully-trained systems whose extraction performance was close to the highest levels achieved by carefully optimized systems based on hand-written rules. SIFT, the first system described, extracts entities and relations from text. On the sentence level, it combines syntactic and semantic knowledge in a novel way, thus taking advantage of the significant recent progress in statistical parsing and leveraging those techniques for information extraction. Knowledge of English syntax extracted from the Penn Treebank is automatically combined with semantically annotated training material in the target domain that identifies how the entities and relations of interest in the domain are signaled in text. At the message level, the local entities and relations identified within each sentence are then merged, and cross-sentence relations are identified using an additional trained model. The resulting system achieved the second-best score of those participating in the MUC-7 evaluation. The second system described here is the IdentiFinder TM system for locating named entities. This system is a fully-trained, HMM- based model that learns from examples the contextual clues that help to identify names in the text. STATISTICAL EXTRACTION OF ENTITIES AND RELATIONS The SIFT system ("Statistically-derived Information From Text") combines a sentence- level model with message-level processing to merge elements and identify cross-sentence relations. At the sentence level, SIFT employs a unified statistical process to map from words to semantic structures. That is, part-of-speech determination, name-finding, parsing, and …

Content truncated to 500 words