An Approach to Text Corpus Construction which Cuts Annotation Costs and Maintains Reusability of Annotated Data

Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 486–495, Prague, June 2007. c⃝2007 Association for Computational Linguistics An Approach to Text Corpus Construction which Cuts Annotation Costs and Maintains Reusability of Annotated Data Katrin Tomanek Joachim Wermter Udo Hahn Jena University Language & Information Engineering (JULIE) Lab F¨urstengraben 30 D-07743 Jena, Germany {tomanek|wermter|hahn}@coling-uni-jena.de Abstract We consider the impact Active Learning (AL) has on effective and efficient text cor- pus annotation, and report on reduction rates for annotation efforts ranging up until 72%. We also address the issue whether a corpus annotated by means of AL – using a particu- lar classifier and a particular feature set – can be re-used to train classifiers different from the ones employed by AL, supplying alter- native feature sets as well. We, finally, report on our experience with the AL paradigm un- der real-world conditions, i.e., the annota- tion of large-scale document corpora for the life sciences. 1 Introduction The annotation of corpora has become a crucial pre- requisite for NLP utilities which rely on (semi-) su- pervised machine learning (ML) techniques. While stability, by and large, has been reached for tagsets up until the syntax layer, semantic annotations in terms of (named) entities, semantic roles, proposi- tions, events, etc. reveal a high degree of variability due to the inherent domain-dependence of the under- lying tagsets. This diversity fuels a continuous need for creating semantic annotation data anew. Accordingly, annotation activities will persist and even increase in number as HLT is expanding on various technical and scientific domains (e.g., the life sciences) outside the classical general-language newspaper genre. Since the provision of annota- tions is a costly, labor-intensive and error-prone pro- cess the amount of work and time this activity re- quires should be minimized to the extent that corpus data could still be used to effectively train ML-based NLP components on them. The approach we ad- vocate does exactly this and yields reduction gains (compared with standard procedures) ranging be- tween 48% to 72%, without seriously sacrificing an- notation quality. Various techniques to minimize the necessary amount of annotated training material have al- ready been investigated. In co-training (Blum and Mitchell, 1998), e.g., from a small initial set of la- beled data multiple learners mutually provide new training material for each other by labeling unseen examples. Pierce and Cardie (2001) have shown, however, that for tasks which require large numbers of labeled examples – such as most NLP tasks – co- training might be inadequate because it tends to gen- erate noisy data. Furthermore, a well compiled ini- tial training set is a crucial prerequisite for success- ful co-training. As another alternative for minimiz- ing annotation work, active learning (AL) is based on the idea to let the learner have control over the ex- amples to be manually labeled so as to optimize the prediction accuracy. Accordingly, AL aims at select- ing those examples with high utility for the model. AL (as …

Content truncated to 500 words