Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future

Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future Jan-Christoph Klie∗ Ubiquitous Knowledge Processing Lab Department of Computer Science Technical University of Darmstadt www.ukp.tu-darmstadt.de Bonnie Webber School of Informatics, University of Edinburgh Iryna Gurevych UKP Lab / TU Darmstadt Annotated data is an essential ingredient in natural language processing for training and evaluating machine learning models. It is therefore very desirable for the annotations to be of high quality. Recent work, however, has shown that several popular datasets contain a surprising number of annotation errors or inconsistencies. To alleviate this issue, many meth- ods for annotation error detection have been devised over the years. While researchers show that their approaches work well on their newly introduced datasets, they rarely compare their methods to previous work or on the same datasets. This raises strong concerns on methods’ general performance and makes it difficult to assess their strengths and weaknesses. We therefore reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets for text classification as well as token and span labeling. In addition, we define a uniform evaluation setup including a new formalization of the annotation error detection task, evaluation protocol, and general best practices. To facilitate future research and reproducibility, we release our datasets and implementations in an easy-to-use and open source software package.1 1. Introduction Annotated corpora are an essential component in many scientific disciplines, including natural language processing (NLP) (Gururangan et al. 2020; Peters, Ruder, and Smith ∗ Corresponding author. 1 https://github.com/UKPLab/nessie. Action Editor: Nianwen Xue. Submission received: 7 June 2022; revised version received: 29 August 2022; accepted for publication: 23 September 2022. https://doi.org/10.1162/coli_a_00464 © 2022 Association for Computational Linguistics Published under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license Computational Linguistics Volume 49, Number 1 2019), linguistics (Haselbach et al. 2012), language acquisition research (Behrens 2008), and the digital humanities (Schreibman, Siemens, and Unsworth 2004). Corpora are used to train and evaluate machine learning models, to deduce new knowledge, and to suggest appropriate revisions to existing theories. Especially in machine learning, high- quality datasets play a crucial role in advancing the field (Sun et al. 2017). It is often taken for granted that gold standard corpora do not contain errors—but alas, this is not always the case. Datasets are usually annotated by humans who can and do make mistakes (Northcutt, Athalye, and Mueller 2021). Annotation errors can even be found in corpora used for shared tasks such as CONLL-2003 (Tjong Kim Sang and De Meulder 2003). For instance, Durban is annotated there as PER (person) and S.AFRICA as MISC (miscellaneous), but both should be annotated as LOC (location). Gold standard annotation is also subject to inconsistency, where words or phrases that are intended to refer to the same type of thing (and so should be labeled in the same way) are nevertheless assigned different labels (see, e.g., Hollenstein, Schneider, and Webber 2016). For example, in CONLL-2003, when Fiorentina was used to refer to the local football …

Content truncated to 500 words