The ACL RD-TEC 2.0: A Language Resource for Evaluating Term Extraction and Entity Recognition Methods

The ACL RD-TEC 2.0: A Language Resource for Evaluating Term Extraction and Entity Recognition Methods Behrang QasemiZadeh† and Anne-Kathrin Schumann‡ †DFG Collaborative Research Centre 991, Heinrich-Heine University D¨usseldorf ‡Department of Applied Linguistics, Translation and Interpreting, Saarland University zadeh@phil.hhu.de, annek schumann@gmx.de Abstract This paper introduces the ACL Reference Dataset for Terminology Extraction and Classification, version 2.0 (ACL RD-TEC 2.0). The ACL RD-TEC 2.0 has been developed with the aim of providing a benchmark for the evaluation of term and entity recognition tasks based on specialised text from the computational linguistics domain. This release of the corpus consists of 300 abstracts from articles in the ACL Anthology Reference Corpus, published between 1978–2006. In these abstracts, terms (i.e., single or multi-word lexical units with a specialised meaning) are manually annotated. In addition to their boundaries in running text, annotated terms are classified into one of the seven categories method, tool, language resource (LR), LR product, model, measures and measurements, and other. To assess the quality of the annotations and to determine the difficulty of this annotation task, more than 171 of the abstracts are annotated twice, independently, by each of the two annotators. In total, 6,818 terms are identified and annotated, resulting in a specialised vocabulary made of 3,318 lexical forms, mapped to 3,471 concepts. We explain the development of the annotation guidelines and discuss some of the challenges we encountered in this annotation task. Keywords: Terminology, Term Extraction, Term Classification, Entity Recognition, Evaluation Corpus, Language Resource. 1. Introduction Terminology mining methods are the cornerstone of mod- ern information systems. These methods are concerned with the automatic analysis of languages for special pur- poses—for example, to facilitate knowledge acquisition from text, to enhance the interoperability when commu- nicating knowledge between communities or across lan- guages, to provide information summaries, and so on. One way or another, these methods deal with the extraction of lexical items known as terms. Linguistically, a term is a lexical unit that carries a spe- cialised meaning in a particular context (e.g., see Faber and Rodr´ıguez (2012)). According to their lexical forms, terms are either simple (made of one word/token) or com- plex (i.e., multi-word units). The extraction of terms has been addressed under various names, different condi- tions (e.g., by restrictions applied to their input and out- put), and for colourful purposes. The most familiar exam- ples, perhaps, are methods for automatic term extraction (ATR), key-phrase extraction, and entity extraction—see QasemiZadeh (2015, chap. 3) for an explanation of similar- ities and differences between these methods. The evalua- tion of all these methods, however, has been tackled in a similar fashion. The black-box, data-driven evaluation process—as initiated in the series of message understanding conferences (MUC) and still used in semantic evaluation workshops (SemEval), etc.—is the dominant methodology for assessing and com- paring the performance of term extraction methods.1 MUC- like evaluation systems consist of two major components. The first component is a gold dataset: that is, a collec- tion of manually annotated text. The second component is a collection …

Content truncated to 500 words