Plans for a Task-Oriented Evaluation of Natural Language Understanding Systems

PLANS FOR A TASK-ORIENTED EVALUATION OF NATURAL LANGUAGE UNDERSTANDING SYSTEMS Beth M. Sundheim Naval Ocean Systems Center, Code 444 San Diego, CA 92152-5000 ABSTRACT A plan is presented for evaluating natural language processing (NLP) systems that have focused on the issues of text understanding as exemplified in short texts from military messages. The plan includes definition of bodies of text to use as development and test data, namely the narrative lines from one type of naval message, and definition of a simulated database update task that requires NLP systems to fill a template with information found in the texts. Documentation related to the naval messages and examples of filled templates have been prepared to assist NLP system developers. It is anticipated that developers of a number of different NLP systems will participate in the evaluation and will meet afterwards to present and interpret the results and to critique the test design. INTRODUCTION This project undertakes to provide meaningful measures of progress in the field of natural language processing (NLP). In particular, it is intended to result in definition of a theory- and implementation-independent test of the text analysis capabilities of text understanding systems that analyze short (paragraph-length) texts taken from military messages. The test is task-oriented in order to facilitate assessment of the general state of the art and provide a meaningful basis for comparing notes across systems. This design would seem to have two major problems, however: the reduction of a system's capabilities to a simple quantification of right versus wrong answers, and the lack of desired focus on understanding capabilities versus application capabilities. It is claimed, however, that if the task performance is recorded on development data as well as test data and is repeated on the test data after updates are made, additional insights can be gained into a sytem's robustness, breadth and depth of coverage, and potential for handling novel text. A measurement of utility can be gained as well, by measuring performance on the original task, versus performance using a version of the inputs in which punctuation and spelling errors, highly elliptical constructions and sublanguage constructions have been eliminated. These additional measurements open up the black box to some extent, providing information that far exceeds what would be obtainable from a single measurement of performance on the test data in a blind test. Also, despite the fact that the NLP systems are treated as black boxes, the evaluation should provide significant insights into their understanding versus application capabilities, because successful performance of the task does not require that back end modules contribute substantive information to the template fills. For example, the template fills do not require that any computations be performed on the data. This aspect of the test design is another way in which the black box has been opened up or narrowed down to increase the meaningfulness of the results. It is important, however, to recognize that the test is applicable only to "complete" and non-interactive systems, ones that are capable of accepting …

Content truncated to 500 words