Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2591–2597 June 6–11, 2021. ©2021 Association for Computational Linguistics 2591 Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning Tommaso Fornaciari Universit`a Bocconi fornaciari@unibocconi.it Barbara Plank IT University of Copenhagen bapl@itu.dk Alexandra Uma Queen Mary University a.n.uma@qmul.ac.uk Dirk Hovy Universit`a Bocconi dirk.hovy@unibocconi.it Silviu Paun Queen Mary University s.paun@qmul.ac.uk Massimo Poesio Queen Mary University m.poesio@qmul.ac.uk Abstract Supervised learning assumes that a ground truth label exists. However, the reliability of this ground truth depends on human anno- tators, who often disagree. Prior work has shown that this disagreement can be helpful in training models. We propose a novel method to incorporate this disagreement as informa- tion: in addition to the standard error com- putation, we use soft labels (i.e., probability distributions over the annotator labels) as an auxiliary task in a multi-task neural network. We measure the divergence between the pre- dictions and the target soft labels with several loss-functions and evaluate the models on vari- ous NLP tasks. We find that the soft-label pre- diction auxiliary task reduces the penalty for errors on ambiguous entities and thereby miti- gates overfitting. It significantly improves per- formance across tasks beyond the standard ap- proach and prior work. 1 Introduction Usually, the labels used in NLP classification tasks are produced by sets of human annotators. As dis- agreement between annotators is common, many methods aggregate the different answers into a supposedly correct one (Dawid and Skene, 1979; Carpenter, 2008; Hovy et al., 2013; Raykar et al., 2010; Paun et al., 2018; Ruiz et al., 2019). How- ever, the aggregated labels obtained in this way mask the world’s real complexity: instances can be intrinsically ambiguous (Poesio and Artstein, 2005; Zeman, 2010; Plank et al., 2014; Pavlick and Kwiatkowski, 2019), or so challenging to evaluate that considerable disagreement between different annotators is unavoidable. In those cases, it is rea- sonable to wonder whether the ambiguity is indeed harmful to the models or whether it carries valuable information about the relative difficulty of each in- stance (Aroyo and Welty, 2015). Several authors followed that intuition, trying ways to incorporate the information about the level of annotator agree- ment in their models (Sheng et al., 2008; Plank et al., 2014, 2016; Jamison and Gurevych, 2015; Rodrigues and Pereira, 2018; Lalor et al., 2017). Usually, Deep Learning models compute the er- ror as the divergence between the predicted label distribution and a one-hot encoded gold distribution (i.e., nothing but the gold label has any probability mass). However, for complex tasks, this binary black-and-white notion of truth is not plausible and can lead to overfitting. Instead, we can use a more nuanced notion of truth by comparing against soft labels: we collect the probability distributions over the labels given by the annotators, rather than using one-hot encodings with a single correct label. To measure the divergence between probability distri- butions, we can use well-known measures like the Kullback-Leibler …

Content truncated to 500 words