Would you describe a leopard as yellow? Evaluating crowd-annotations with justified and informative disagreement

Proceedings of the 28th International Conference on Computational Linguistics, pages 4798–4809 Barcelona, Spain (Online), December 8-13, 2020 4798 Would you describe a leopard as yellow? Evaluating crowd-annotations with justified and informative disagreement Pia Sommerauer∗ Antske Fokkens∗† ∗Computational Lexicology and Terminology Lab, Vrije Universiteit Amstersterdam De Boelelaan 1105, 1081HV Amsterdam †Department of Mathematics and Computer Science, Eindhoven University of Technology P.O.Box 513, MetaForum 5600 MB Eindhoven {pia.sommerauer,antske.fokkens,p.t.j.m.vossen}@vu.nl Piek Vossen∗ Abstract Semantic annotation tasks contain ambiguity and vagueness and require varying degrees of world knowledge. Disagreement is an important indication of these phenomena. Most traditional eval- uation methods, however, critically hinge upon the notion of inter-annotator agreement. While alternative frameworks have been proposed, they do not move beyond agreement as the most im- portant indicator of quality. Critically, evaluations usually do not distinguish between instances in which agreement is expected and instances in which disagreement is not only valid but de- sired because it captures the linguistic and cognitive phenomena in the data. We attempt to overcome these limitations using the example of a dataset that provides semantic representations for diagnostic experiments on language models. Ambiguity, vagueness, and difficulty are not only highly relevant for this use-case, but also play an important role in other types of seman- tic annotation tasks. We establish an additional, agreement-independent quality metric based on answer-coherence and evaluate it in comparison to existing metrics. We compare against a gold standard and evaluate on expected disagreement. Despite generally low agreement, annotations follow expected behavior and have high accuracy when selected based on coherence. We show that combining different quality metrics enables a more comprehensive evaluation than relying exclusively on agreement. 1 Introduction Would you say leopards are yellow? Most likely, some people would while others would not. Both inter- pretations are valid, as the interpretation depends on a person’s boundaries for the properties ‘yellow’ and ‘brown’. Selecting only one judgment would disregard the vagueness of the expression, a phenomenon at the heart of lexical semantics. At the same time, most people would probably agree that wine can be red without having to think about it. A high number of semantic annotation tasks is characterized by unclear, difficult, ambiguous and vague examples. Annotation, in particular when distributed among a crowd, has the potential of capturing different interpretations, conceptualizations and perspectives and can thus provide highly relevant semantic information. Existing evaluation and label extraction methods, however, still heavily rely on agreement between annotators, which implies a single correct interpreta- tion. Finished datasets rarely provide indications about difficulty and ambiguity on the level of annotated units. The explanatory power of NLP experiments that aim to evaluate or analyze models depends on the informativeness of the data. This is particularly relevant for experiments which specifically aim to un- derstand models better, such as the tradition of diagnostic experiments (Belinkov and Glass, 2019).Tra- ditional error analyses could also benefit substantially from test sets which contain information about phenomena with a likely impact on model performance. Furthermore, knowing whether model-errors are similar to human …

Content truncated to 500 words