Criteria for Measuring Term Recognition

Criteria for Measuring Term Recognition Andy Lauriston Department of Languages and Linguistics University of Manchester Institute of Science and Technology P.O. Box 88 Manchester M60 1QD United Kingdom andyl@ccl.umist.ac.uk Abstract This paper qualifies what a true term- recognition systems would have to recog- nize. The exact bracketing of the maximal termform is then proposed as an achieve- able goal upon which current system per- formance should be measured. How recall and precision metrics are best adapted for measuring term recognition is suggested. 1 Introduction In recent years, the automatic extraction of terms from running text has become a subject of grow- ing interest. Practical applications such as dictio- nary, lexicon and thesaurus construction and main- tenance, automatic indexing and machine transla- tion have fuelled this interest. Given that concerns in automatic term recognition are practical, rather than theoretical, the lack of serious performance measurements in the published literature is surpris- ing. Accounts of term-recognition systems sometimes consist of a purely descriptive statement of the ad- vantages of a particular approach and make no at- tempt to measure the pay-off the proposed approach yields (David, 1990). Others produce partial fig- ures without any clear statement of how they are derived (Otman, 1991). One of the best efforts to quantify the performance of a term-recognition sys- tem (Smadja, 1993) does so only for one processing stage, leaving unassessed the text-to-output perfor- mance of the system. While most automatic term-recognition systems developed to date have been experimental or in- house ones, a few systems like TermCruncher (Nor- mand, 1993) are now being marketed. Both the developers and users of such systems would benefit greatly by clearly qualifying what each system aims to achieve, and precisely quantifying how closely the system comes to achieving its stated aim. Before discussing what a term-recognition system should be expected to recognize and how perfor- mance in recognition should be measured, two un- derlying premises should be made clear. Firstly, the automatic system is designed to recognize seg- ments of text that, conventionally, have been man- ually identified by a terminologist, indexer, lexicog- rapher or other trained individual. Secondly, the performance of automatic term-recognition systems is best measured against human performance for the same task. These premises mean that for any given application - terminological standardization and vo- cabulary compilation being the focus here - it is pos- sible to measure the performance of an automatic term-recognition system, and the best yardstick for doing so is human performance. Section 2 below draws on the theory of terminol- ogy in order to qualify what a true term-recognition system must achieve and what, in the short term, such systems can be expected to achieve. Section 3 specifies how the established ratios used in infor- mation retrieval - recall and precision - can best be adapted for measuring the recognition of single- and multi-word noun terms. 2 What is to be Recognized? Depending upon the meaning given to the expres- sion "term recognition", it can be viewed as either a rather …

Content truncated to 500 words