A Formal Basis for Performance Evaluation of Natural Language Understanding Systems

A Formal Basis for Performance Evaluation of Natural Language Understanding Systems Giovanni Guida 1 and Giancarlo Mauri 2 Istituto di Matematica, Informatica e Sistemistica Universit~ di Udine Udine, Italy The task of evaluating the performance of a natural language understanding system, despite its largely recognized relevance, is still poorly defined. It mostly relies on intuitive reasoning and lacks a sound theoretical foundation. This paper sets a formal and quantita- tive proposal for this task. In particular, a measure of performance that allows the basic input-output characteristics of a system to be evaluated is introduced first at an abstract level. The definition of concrete measures is then obtained by assigning actual values to the functional parameters of the abstract definition; some particular cases are shown and discussed in detail. Finally, the task of measuring performance in practice is considered, and a model for experimental performance evaluation is presented. Comparison with related works is also briefly discussed; open problems and promising directions for future research are outlined. A limited case study experimentation with the model proposed is presented in the appendix. 1. Introduction Research on natural language processing has recently been featured by the design and implementation of a number of experimental systems. Recent survey re- ports (Waltz 1977, Kaplan 1982) mention more than one hundred items among the most successful and relevant systems in the classical application fields of data base inquiry, machine translation, question an- swering, and man-machine interfacing. This trend is not surprising in the context of re- search whose specific aim is that of providing automat- ed tools for the understanding or translating of natural languages; but it is also evident even in natural lan- guage research with a more theoretical flavour. The successful construction of a good performing system is 1 Address: Prof. Giovanni Guida Dipartimento di Elettronica Politecnico di Milano P.zza Leonardo da Vinci, 32 i-20133 MILANO, Italy Also with Milan Polytechnic Artificial Intelligence Project, Milano, Italy. 2 Also with lstituto di Cibernetica, Universita di Milano, Mila- no, Italy. in fact often considered as the most evident proof of the validity of a theory, and, therefore, designing run- ning systems is routine, and even sometimes the spe- cific goal of several researchers. The task of evaluating the performance of a given system and that of comparing the behaviour of differ- ent systems appears, therefore, to be a fundamental issue. Despite its large recognized relevance (Woods 1977, Tennant 1980), measuring the performance of a system for natural language processing is still poorly defined. It mostly relies on intuitive reasoning and lacks a sound theoretical foundation. As Tennant clearly points out (1980), there is a nearly complete absence of meaningful evaluation in current natural language processing research. This leaves several cru- cial questions unanswered: • What is the relevance and value of obtained results? • How general are the proposed solutions? • How do they compare with other proposals? • What problems are still open? • What directions have to be followed? • What issues are to be …

Content truncated to 500 words