Evaluating Term Extraction

International Conference RANLP 2009 - Borovets, Bulgaria, pages 299–304 Evaluating term extraction Adeline Nazarenko Ha¨ıfa Zargayouna Laboratoire d’Informatique de Paris-Nord Laboratoire d’Informatique de Paris-Nord CNRS UMR 7030 - Univ. Paris 13 CNRS UMR 7030 - Univ. Paris 13 99, avenue J.B. Cl´ement 99, avenue J.B. Cl´ement F-93430 Villetaneuse F-93430 Villetaneuse nazarenko@lipn.univ-paris13.fr haifa@lipn.univ-paris13.fr Abstract In contrast with other NLP tasks, only few and limited evaluation challenges have been carried out for terminology acquisition. It is nevertheless important to assess the progress made, the qual- ity and limitations of terminological tools. This paper argues that it is possible to define evalua- tion protocols for tasks as complex as computa- tional terminology. We focus on the core task of term extraction for which we propose evaluation metrics. We take into account the specificity of computational terminology, the complexity of its outputs, the application, the user’s role and the absence of well-established gold standard. Keywords Term extraction, evaluation, terminological distance. 1 Introduction Stemming from traditional terminology and natu- ral language processing (NLP), computational ter- minology aims at building automatically or semi- automatically terminological resources from acquisi- tion corpora. The growing needs in information man- agement and localization make it more and more nec- essary to assist and automate terminological tasks. A lot of terminological tools have been developed since the early research works of the 90s and many content management companies now rely on them [20]. However, despite the progress made, it remains diffi- cult to get a clear idea of the maturity of computa- tional terminology and to compare the proposed ap- proaches. Unlike many other NLP fields, only few ef- fort has been made to set up an evaluation protocol adapted to the specificity of terminological tasks. We nevertheless argue that an evaluation is possi- ble in computational terminology and that defining a clear and consensual evaluation protocol would bene- fit to the whole field. This paper focuses on techni- cal aspects of evaluation putting aside the ergonomic and software aspects. We propose a comparative and application-independent evaluation protocol for mono- lingual term extraction as a first step towards more global and application-oriented evaluations. Sections 2 and 3 review the first experiments that have been carried out for evaluating terminological tools and the difficulties that such evaluations raise. Sections 4 and 5 present our proposal: a protocol for evaluating term extractors and the specific metrics on which it relies. Section 6 describes experiments for meta-evaluating the proposed metrics. 2 State of the art Various experiments have been made to evaluate ter- minological tools. Some were technologically oriented and took the form of evaluation challenges while others put focus on the application context. 2.1 Evaluation challenges Traditionally, evaluation challenges aim at evaluating a set of systems on a specific task and for a common data set. The systems are compared to each other or wrt. a common data set. This enables the ranking of the systems for the specified task. The first evaluation challenges proposed interesting protocols. The NTCIR1 initiative was launched in …

Content truncated to 500 words