Towards Better NLP System Evaluation

TOWARDS BETTER NLP SYSTEM EVALUATION Karen Sparck Jones Computer Laboratory, University of Cambridge New Museums Site, Pembroke Street, Cambridge CB2 3QG sparckjones@cl.cam.ac.uk ABSTRACT This paper considers key elements of evaluation methodol- ogy, indicating the many points involved and advocating an unpacking approach in specifying an evaluation remit and de- sign. Recognising the importance of both environment vari- ables and system parameters leads to a grid organisation for tests. The paper illustrates the application of these notions through two examples. 1. Introduction There is a manifest need for a good evaluation method- ology for (S&)NLP systems. This paper presents such a methodology, indicating its key elements under the head- ings of performance factors, evaluation gauges, test data, and assessment strategy, and illustrating its application. The paper is based on Galliers and Sparck Jones (1993), which offers a comprehensive analysis of what is involved and needed in NLP evaluation, supplemented by an ex- tensive review of methodologies that have already been proposed or applied, and also refers to evaluation ex- perience in the neighbouring information retrieval (IR) field. 2. Background There is generally growing interest and activity in NLP system evaluation, stimulated partly by the fact that NLP technology is now solid enough to be of interest for real applications and partly by the (D)ARPA ini- tiatives (cf Thompson, 1992; HLT1; MUC; S&NLW; TREC). But evaluation practice is still uneven, and sound methodologies need to be developed, both for lab- oratory experiments and working system investigations. There are many difficult matters to be considered in NLP evaluation, and some significant points seem not to be sufficiently recognised. Thus there are dangers in seeking to apply 'closed problem' approaches, exemplified by the use of 'word error rate', outside their legitimate bounds; in attaching prime importance to internal objects like parse trees; or in reifying important but ultimately arbi- trary processing devices, for instance specific sets of case labels. The IR community's hard-won experience shows there are no easy ways of evaluating systems, no magic numbers encapsulating performance, no 'core' functions that can be pursued far in isolation, no fixed meaning- representation devices any system must have. Further, the lesson of IR is that discourse processing (in this case indexing) is only as good as the use made of it, which is necessarily on individual occasions, so it is only pos- sible to infer, over many such occasions, what average performance will or can be designed to be. The moti- vation for a better evaluation methodology is thus more reliable inference, implying both a diligent attempt to lay out the premises but also, given the many factors involved in NLP systems and their resistance to precise specification, that the inferences drawn may still not be very reliable. The essential problem of NLP evaluation is that the evaluator has to advance between Scylla and Charyb- dis. Evaluation may be end-to-end for systems devoted to NLP, end-to-end for hybrid ones with both NLP and non-NLP subsystems, or non end-to-end, focused on some component of an NL …

Content truncated to 500 words