Evaluating Natural Language Generated Database Records

Evaluating Natural Language Generated Database Records Rita McCardell Department of Defense Fort Meade, Maryland 20755 ABSTRACT With the onslaught of various natural language pro- cessing (NLP) systems and their respective applica- tions comes the inevitable task of determining a way in which to compare and thus evaluate the output of these systems. This paper focuses on one such evalua- tion technique that originated from the text understand- ing system called Project MURASAKI. This evaluation technique quantitatively and qualitatively measures the match (or distance) from the output of one text under- standing system to the expected output of another. Introduction Project MURASAKI The purpose of Project MURASAKI is to develop a foreign language text understanding system that will demonstrate the extensibility of message understanding technology3 In its current design, Project MURASAKI will process Spanish and Japanese text and extract in- formation in order to generate records in both natural language databases, respectively. The fields within these database records will contain a natural language phrase or expression in that respective language. The domain of Project MURASAKI is the disease AIDS. The associated software system will include a general domain model of AIDS in the knowledge base. Within this model, there will be five subdomains: incidence reports records the occurrence of AIDS and HIV infection in countries and regions, among various populations, testing policies covers measures to test groups for AIDS, campaigns describes measures adopted to combat AIDS, new technologies lists new equipment and material used in detecting and preventing AIDS, and 1Thus, it is no_...t to be confused as a message undel~tanding project, but rather a multi-paragraph (i.e., text) understanding project [51. AIDS research details the various vaccines and treat- ments that are being developed to prevent AIDS. The subdomains of incidence reports, testing poli- cies and campaigns are found in the Spanish text while the topics of incidence reports, new technologies and AIDS research are covered in the Japanese text. Project MURASAKI will demonstrate a sufficient level of full text understanding to be able to identify the existence of factual information within either a given Spanish or Japanese text that belongs within a partic- ular Spanish or Japanese language database. Then, it will determine what information in that text constitutes a single record in the selected database. The balance of this paper will focus on the evaluation technique: why it was chosen, some basic assumptions underlying it, as well as the design and application of this technique. To illustrate various technical points of this technique, examples will be given using text excerpted from the Spanish AIDS corpus and its associated (gener- ated) Spanish database records. Appendix A contains a sample Spanish AIDS text (Text #124) and its English translation. 2 Appendix B contains a record from the Incidence Reporting database that was generated from Text #124. Similarly, Appendix C contains a record from the Testing Policies database that was also gener- ated from Text #124. The Need for a Black Box Given the overall design of this foreign language text understanding …

Content truncated to 500 words