Validity, Agreement, Consensuality and Annotated Data Quality

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2940–2948 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2940 Validity, Agreement, Consensuality and Annotated Data Quality Ana¨elle Baledent, Yann Mathet, Antoine Widl¨ocher, Christophe Couronne, Jean-Luc Manguin Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC, 14000 Caen, FRANCE {anaelle.baledent, yann.mathet, antoine.widlocher, christophe.couronne, jean-luc.manguin}@unicaen.fr Abstract Reference annotated (or gold-standard) datasets are required for various common tasks such as training for machine learning systems or system validation. They are necessary to analyse or compare occurrences or items annotated by experts, or to compare objects resulting from any computational process to objects annotated (selected and characterized) by experts. But, even if reference annotated gold-standard corpora are required, their production is known as a difficult problem, from both a theoretical and practical point of view. Many studies devoted to these issues conclude that multi-annotation is most of the time a necessity. Measuring the inter-annotator agreement, which is required to check the reliability of data and the reproducibility of an annotation task, and thus to establish a gold standard, is another thorny problem. Fine analysis of available metrics for this specific task then becomes essential. Our work is part of this effort and more precisely focuses on several problems, which are rarely discussed, although they are intrinsically linked with the interpretation and the evaluation of metrics. In particular, we focus here on the complex relations between agreement and reference (of which agreement among annotators is supposed to be an indicator), and the emergence of a consensus. We also introduce the notion of consensuality as another relevant indicator. Keywords: Annotations, Agreement, Validity, Consensuality 1. Introduction There is a broad consensus concerning the need for gold-standard annotated datasets, and many annotation campaigns aim at producing such datasets. Especially in the Natural Language Procession (NLP) community we come from, they provide a crucial element to study a phenomenon, to evaluate or to train a system devoted to the detection or the analysis of that phenomenon. How- ever, the creation of such datasets is accompanied by theorectical and practical issues, especially when an- notated phenomena are complex and require fine inter- pretation. When it is considered as impossible to reach directly, the establishment of the reference is therefore in practice often postponed, in favor of a confrontation of multiple points of view on the same data. Indeed, it is assumed that a significant agreement among anno- tators is a necessary condition for the emergence of a reference. A more debatable hypothesis assumes that such an agreement is also a sufficient condition. In order to clarify the methodological framework in which reference data are produced, communities pro- moting data-centric approaches tend to support works directly devoted to these questions (Aroyo and Welty, 2015; Oortwijn et al., 2021). For example, in the NLP community, we can mention (Fort, 2016) or (Mathet et al., 2015), where it appears that measuring agree- ment among annotators – a necessary prerequisite to the establishment of a reference – is …

Content truncated to 500 words