Errator: a Tool to Help Detect Annotation Errors in the Universal Dependencies Project

ERRATOR: a Tool to Help Detect Annotation Errors in the Universal Dependencies Project Guillaume Wisniewski LIMSI, CNRS, Univ. Paris-Sud, Universit´e Paris Saclay, 91 403 Orsay, France guillaume.wisniewski@limsi.fr Abstract Enforcing guidelines compliance is today one of the main challenge faced by the Universal Dependencies project. This work introduces ERRATOR, a set of tools implementing the annotation variation principle that can be used to help annotators find and correct errors in the different layers of annotations of UD treebanks. The results of a first annotation campaign that used ERRATOR to correct and harmonize the annotations of the different French corpora are also described. Keywords: Universal Dependencies, Annotation Coherence, Evaluation 1. Introduction The Universal Dependencies project (Nivre et al., 2017) aims at developing cross-linguistically consistent treebank annotations for a wide array of languages. Each treebank contains raw text, sentence and word segmentation, PoS tags, dependency relations and in many cases lemmas and morphological features. In its latest release, the UD project gathers 70 treebanks covering 50 languages. Many of these corpora result from a manual or semi-automatic transformation from existing de- pendency or constituent treebanks into the UD formalism (see, for instance, (Bosco et al., 2013) or (Lipenkova and Souˇcek, 2014) for a description of such transformations). Because many treebanks have been annotated and/or con- verted independently by different groups, the risk of inco- herence and errors in the application of annotation guide- lines is increased. There may indeed be several sources of errors in the produced annotations: in addition to the di- vergences in the theoretical linguistic principles that gov- erned the design of the original annotation guidelines, er- rors may also result from automatic (pre-)processes, human post-editing, or human annotation. As a matter of fact, several works have recently pointed out that different treebanks for the same language are not consistently annotated even though they should follow the same guidelines (Aufrant et al., 2017; Vilares and G´omez- Rodr´ıguez, 2017). More generally, (Wisniewski et al., 2014) has shown that, in spite of common annotation guide- lines, the main bottleneck in cross-lingual transfer is the difference in the annotation conventions across corpora and languages. Enforcing guidelines compliance to improve annotation quality is therefore one of the main challenge faced by the UD project today: as it is, performance achieved on UD corpora, especially in a cross-lingual or cross-corpus setting may be underestimated and this drop may results mainly from divergences in annotations. This work de- scribes ERRATOR, a set of tools implementing the anno- tation variation principle that has been proposed to detect errors in PoS annotations (van Halteren, 2000; Dickinson and Meurers, 2003) and syntactic annotations (Dickinson and Meurers, 2005; Boyd et al., 2008). We propose an ex- tension of this principle to word segmentation making ER- RATOR able to help annotators find and correct errors in the different layers of annotations of the UD corpora. The rest of this paper is organized as follows: we will first explain how the annotation variation principle can be used to identify potential annotation errors …

Content truncated to 500 words