Quality and Efficiency of Manual Annotation: Pre-annotation Bias

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2909–2918 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2909 Quality and Efficiency of Manual Annotation: Pre-annotation Bias Marie Mikulov´a, Milan Straka, Jan ˇStˇep´anek, Barbora ˇStˇep´ankov´a, Jan Hajiˇc Institute of Formal and Applied Linguistics Computer Science School, Faculty of Mathematics and Physics, Charles University, Prague {mikulova,straka,stepanek,stepankova,hajic}@ufal.mff.cuni.cz Abstract This paper presents an analysis of annotation using an automatic pre-annotation for a mid-level annotation complexity task - dependency syntax annotation. It compares the annotation efforts made by annotators using a pre-annotated version (with a high-accuracy parser) and those made by fully manual annotation. The aim of the experiment is to judge the final annotation quality when pre-annotation is used. In addition, it evaluates the effect of automatic linguistically-based (rule-formulated) checks and another annotation on the same data available to the annotators, and their influence on annotation quality and efficiency. The experiment confirmed that the pre-annotation is an efficient tool for faster manual syntactic annotation which increases the consistency of the resulting annotation without reducing its quality. Keywords: annotation, syntax, pre-annotation bias, annotation efficiency, annotation quality, inter-annotator agreement 1. Introduction Developing high-quality annotated corpora represent- ing natural language phenomena that can be used by machine learning tools or explored by linguists solving various research tasks is time-consuming and expen- sive. This paper presents results of a carefully designed experiment that shed some light on the issue of effi- ciency vs. quality of manual syntactic annotation. For this purpose, we used manually annotated parts of the Prague Dependency Treebank - Consolidated, PDT-C in sequel (Hajiˇc et al., 2020)1 under various experimen- tal set-ups: annotation with no support (“from-scratch mode”), annotation supported by the use of automatic pre-annotation (using a high-accuracy parser), avail- ability of other annotation on the same data, and “on- line” checking rules (implemented in such a way that the annotator can check the manual annotation or man- ual post-editing immediately). The experiments focus on dependency syntax annotation: each token of a sen- tence is assigned its head and specific type of the de- pendency relation. With the experiment, we aim to answer three main re- search questions, the first of which is crucial: • Does the quality of manual annotation remain acceptably high if pre-annotation is used? How much the manual correction of automatic pre- annotation increases quality? The concern is that annotators might tend to simply go along with the pre-annotation, which would lead to a lower qual- ity than they could produce by a fully manual an- notation (cf. similar question in Rehbein et al. (2009)). • What effect do other available tools (checking rules, other annotation on the same data) have on 1https://hdl.handle.net/11234/1-3185 the quality and efficiency of the annotation and to what extent? • Which set-up is most useful for manual de- pendency syntax annotation? We want to find out how to best prepare the annotation environ- ment and workflow for the syntactic annotation of 2,000,000 tokens in the …

Content truncated to 500 words