Modeling the Complexity of Manual Annotation Tasks: a Grid of Analysis

Proceedings of COLING 2012: Technical Papers, pages 895–910, COLING 2012, Mumbai, December 2012. Modeling the Complexity of Manual Annotation Tasks: a Grid of Analysis Karën Fort1,2 Adeline Nazarenko1 Sophie Rosset3 (1) LIPN, Université Paris 13 & CNRS, 99 av. J.B. Clément, 93430 Villetaneuse (2) LORIA, Campus Scientifique, 54506 Vandœuvre-lès-Nancy (3) LIMSI-CNRS Rue John von Neumann, Université Paris-Sud 91403 Orsay karen.fort@loria.fr, adeline.nazarenko@lipn.univ-paris13.fr, sophie.rosset@limsi.fr ABSTRACT Manual corpus annotation is getting widely used in Natural Language Processing (NLP). While being recognized as a difficult task, no in-depth analysis of its complexity has been performed yet. We provide in this article a grid of analysis of the different complexity dimensions of an annotation task, which helps estimating beforehand the difficulties and cost of annotation campaigns. We observe the applicability of this grid on existing annotation campaigns and detail its application on a real-world example. KEYWORDS: manual corpus annotation, annotation campaign management, annotation cam- paign cost estimate. 895 1 Introduction With the development of NLP applications, the annotation campaigns are becoming more numerous and more varied. Annotated corpora are used for acquiring knowledge as well as for testing theories, models and tools. They can be directly used in end-applications or for specific internal tasks. Manual annotation is actually a widespread practice in NLP. It consists in adding labels of linguistic nature or reflecting the usage of NLP technologies on some oral or written discourse. This corresponds to a great diversity of phenomena, as these annotations vary in nature (phonetic, morpho-syntactic, semantic or task-oriented labels), in the range they cover (they can concern a couple of characters, a word, a paragraph or a whole text), in their degree of coverage (all the text is annotated or only a part of it) and in their form (atomic value, complex feature structures or relations and even cross-document alignment relations). It has been a long road since the big pioneer annotation campaigns like the Penn Treebank (Marcus et al., 1993), but one problem remains: manual annotation is expensive. Various strate- gies have been implemented to reduce or control annotation costs. Tools have been developed to assist and guide the work of annotators. Automatic annotation methods, sometimes based on machine learning, have been introduced to relieve the annotator of the most trivial and repetitive work and to allow him/her to focus on the hardest annotation tasks where human interpretation is critical. For simple tasks, the use of crowdsourcing is developed with the idea of dividing up tedious work and exploiting the number of annotators to compensate for the heterogeneity of their competence and reliability. Significant efforts have also been made to develop evaluation protocols and to measure intra and inter-annotator agreements, which allow for a better control of the quality of the produced annotated data. Despite all this work and the experience gained in annotation, we lack a global picture or an overall methodology to a priori determine the various costs of an annotation campaign (task definition, data preparation, recruitment and management of annotators, annotation itself, quality control, etc.), make …

Content truncated to 500 words