A Gold Standard for Multilingual Automatic Term Extraction from Comparable Corpora: Term Structure and Translation Equivalents

A Gold Standard for Multilingual Automatic Term Extraction from Comparable Corpora: Term Structure and Translation Equivalents Ayla Rigouts Terryn, Véronique Hoste, Els Lefever LT3 Language and Translation Technology Team, Department of Translation, Interpreting and Communication, Ghent University Groot-Brittanniëlaan 45, 9000 Gent, Belgium {ayla.rigoutsterryn, veronique.hoste, els.lefever}@ugent.be Abstract Terms are notoriously difficult to identify, both automatically and manually. This complicates the evaluation of the already challenging task of automatic term extraction. With the advent of multilingual automatic term extraction from comparable corpora, accurate evaluation becomes increasingly difficult, since term linking must be evaluated as well as term extraction. A gold standard with manual annotations for a complete comparable corpus has been developed, based on a novel methodology created to accommodate for the intrinsic difficulties of this task. In this contribution, we show how the effort involved in the development of this gold standard resulted, not only in a tool for evaluation, but also in a rich source of information about terms. A detailed analysis of term characteristics illustrates how such knowledge about terms may inspire improvements for automatic term extraction. Keywords: terminology extraction, gold standard, comparable corpora 1. Introduction Automatic term extraction (ATE) has been a productive and successful field of research within natural language processing, yet the evaluation of ATE remains particularly difficult. The main difficulty of the task lies in the ambiguous nature of terms; there are no objective rules to distinguish terms from non-terms. While there are many definitions or terms found in the literature, such as “words that are assigned to concepts used in the special languages that occur in subject-fields or domain-related texts” (Wright, 1997, p. 13), these definitions, however accurate, aren’t always helpful when deciding whether or not a lexical unit can be considered a term. Not only does this make ATE a challenge, it also poses a problem for the evaluation. The typical way to evaluate such a task is to compare the automatic output against a manually constructed gold standard (GS), in other words, to compare automatic versus human performance. In this way, precision (how many of the automatically extracted terms are correct) and recall (how many of the terms in the text are automatically extracted) can be calculated. To construct this GS, text must be manually annotated. The corpus must be large and domain-specific enough to be used as input for ATE and it must be annotated entirely to calculate recall. Owing to the ambiguous nature of terms and the necessary volume of text, this is an arduous, time consuming task. Moreover, it results in low inter-annotator agreement scores (Rigouts Terryn et al., Submitted), which are supposed to be an indication of the objectivity and quality of the annotations. Recently, research on ATE has shifted from monolingual ATE, to bilingual ATE, first from parallel corpora and currently also from comparable corpora. ATE from comparable corpora (ATECC) attempts not only to recognise terms in a text, but also to find equivalent terms in the different languages of a comparable corpus. Comparable corpora are collections …

Content truncated to 500 words