Establishing Annotation Quality in Multi-label Annotations

Proceedings of the 29th International Conference on Computational Linguistics, pages 3659–3668 October 12–17, 2022. 3659 Establishing Annotation Quality in Multi-Label Annotations Marian Marchal and Merel Scholman and Frances Yung and Vera Demberg Language Science and Technology / Saarland University {marchal, m.c.j.scholman, frances, vera}@coli.uni-saarland.de Abstract In many linguistic fields requiring annotated data, multiple interpretations of a single item are possible. Multi-label annotations more accurately reflect this possibility. However, allowing for multi-label annotations also af- fects the chance that two coders agree with each other. Calculating inter-coder agreement for multi-label datasets is therefore not triv- ial. In the current contribution, we evaluate different metrics for calculating agreement on multi-label annotations: agreement on the in- tersection of annotated labels, an augmented version of Cohen’s Kappa, and precision, recall and F1. We propose a bootstrapping method to obtain chance agreement for each measure, which allows us to obtain an adjusted agree- ment coefficient that is more interpretable. We demonstrate how various measures affect esti- mates of agreement on simulated datasets and present a case study of discourse relation an- notations. We also show how the proportion of double labels, and the entropy of the label distribution, influences the measures outlined above and how a bootstrapped adjusted agree- ment can make agreement measures more com- parable across datasets in multi-label scenarios. 1 Introduction Annotation efforts have long been characterized by the (implicit or explicit) assumption that there is a single true interpretation for every item, and all other interpretations are incorrect. This has become even more pervasive with the increasing usage of annotations as input for classifiers and other downstream computational tasks. However, the single-truth assumption has been challenged in recent years (e.g., Aroyo and Welty, 2015; Basile et al., 2021), based on counterexamples from many different fields showing that two coders can dis- agree on an annotation and still both be right, due to the subjectivity and complexity of many tasks. We here focus on the field of discourse relation an- notation as a case study, although the insights could be applied to different types of linguistic research (Amidei et al., 2019; Uma et al., 2021). Establishing inter-annotator agreement in sce- nario’s where multiple labels are possible is chal- lenging. In cases where multiple interpretations are possible but coders are restricted to annotate only a single label, it is unclear whether disagreement reflects incorrect interpretations (coder error or is- sues with the coding scheme or training) or item ambiguity. Given an instance where two interpreta- tions are equally likely, assuming that at least one coder inferred both possible readings but only a single label can be annotated, the chance that the coders agree would be only 50%. This does not reflect that there is actually high agreement on the labels. Allowing for multi-label coding can result in more accurate annotations as well as higher agree- ment, and with that most likely higher reliability of a final label. Although two sets of multi-labels can be compared by calculating the difference be- tween their distributions (e.g. …

Content truncated to 500 words