Neal-Montgomery NLP System Evaluation Methodology

NEAL-MONTGOMERY NLP SYSTEM EVALUATION METHODOLOGY Sharon M. Walter Rome Laboratory RL/C3CA Griffiss AFB, NY 13441-5700 walter@ aivax.rl.af.mil ABSTRACT On what basis are the input processing capabilities of Natural Language software judged? That is, what are the capabilities to be described and measured, and what are the standards against which we measure them? Rome Laboratory is currently supporting an effort to develop a concise terminology for describing the linguistic processing capabilities of Natural Language Systems, and a uniform methodology for appropriately applying the terminology. This methodology is meant to produce quantitative, objective profiles of NL system capabilities without requiring system adaptation to a new test domain or text corpus. The effort proposes to develop a repeatable procedure that produces consistent results for independent evaluators. 1. INTRODUCTION An appreciable drawback to current corpus-based (eg., [BBN; 1988], [Flickinger, et al; 1987], [Hendrix, et al; 1976], [Malhotra; 1975]) and task-based (eg., ["Proceedings"; 1991]) methodologies for evaluating Natural Language Processing Systems is the requirement for transportation of the system to a test domain. The expense and time consumption are sizable and, as the port may be minimal or incomplete, the evaluation may be based on a demonstration of less than the full potential of the system. Further, current evaluation methodologies do not fully elucidate NLP system capabilities for possible future applications. Under contract to Rome Laboratory, Dr. Jeannette Neal (Calspan Corporation) and Dr. Christine Montgomery (Language Systems Incorporated) are in the final months of developing an NLP system evaluation methodology that produces descriptive, objective profiles of system linguistic capabilities without a requirement for system adaptation to a new domain. The evaluation methodology is meant to produce consistent results for varied haman users. 1.1. Evaluation Methodology Description Within the Neal-Montgomery NLP System Evaluation Methodology each identified linguistic (lexical, syntactic, semantic, or discourse) feature is first carefully defined and explained in order to establish a standard delimitation of the feature. Illustrative language patterns and sample sentences then guide the human evaluator to the formulation of an input that tests the feature on the NLP system within the system's native domain. Based on clear and specific evaluation criteria for test item inputs, NLP system responses are scored as follows: S: The system successfully met the stated criteria and demonstrated understanding with respect to the feature under test. C: The system responded in a way that was correct (that is, correctly answered the question posed), but the criteria were not met. P: The system responded in a way that was only partially correct. F: The system responded in a way that was incorrect, failing to meet the criteria. N: The system was unable to accept the input or form a response (for example, the system vocabulary lacks appropriate words to complete a test inpu0. Each linguistic feature is tested by more than one methodology item to make sure that results are not based on spurious responses, and each item examines only one as- yet-untested capability, or one as-yet-untested combination of capabilities. Test inputs that are dependent on capabilities previously …

Content truncated to 500 words