UOR at SemEval-2021 Task 12: On Crowd Annotations; Learning with Disagreements to optimise crowd truth

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1303–1309 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 1303 UoR at SemEval-2021 Task 12: On Crowd Annotations; Learning with Disagreements to Optimise Crowd Truth Emmanuel Osei-Brefo, Thanet Markchom , Huizhi Liang University of Reading White Knights, Berkshire, RG6 6AH United Kingdom e.osei-brefo@pgr.reading.ac.uk, t.markchom@pgr.reading.ac.uk huizhi.liang@reading.ac.uk Abstract Crowdsourcing has been ubiquitously used for annotating enormous collections of data. However, the major obstacles to using crowd- sourced labels are noise and errors from non- expert annotations. In this work, two ap- proaches dealing with the noise and errors in crowd-sourced labels are proposed. The first approach uses Sharpness-Aware Mini- mization (SAM), an optimization technique ro- bust to noisy labels. The other approach lever- ages a neural network layer called softmax- Crowdlayer specifically designed to learn from crowd-sourced annotations. According to the results, the proposed approaches can improve the performance of the Wide Residual Net- work model and Multi-layer Perception model applied on crowd-sourced datasets in the im- age processing domain. It also has similar and comparable results with the majority voting technique when applied to the sequential data domain whereby the Bidirectional Encoder Representations from Transformers (BERT) is used as the base model in both instances. 1 Introduction In recent years, there has been some major advance- ment in the use of deep learning for solving arti- ficial intelligence problems in different domains such as sentiment analysis, image classification, natural language inference, speech recognition ob- ject detection. They have also been used in many other numerous cases where human disagreements are encountered such as speech recognition, visual object recognition, object detection and machine translation (Rodrigues and Pereira, 2018). It is however, an essential requirement for deep learn- ing models to utilise labelled data to undertake the representational learning of the underlying datasets. These labelled data are most at times not available and hence the need for humans to manually under- take the labelling of these data becomes a necessity. In recent years, crowd-sourcing has been used in the annotation of large collections of data and has proven to be an efficient and cost-effective means of obtaining labeled data as compared to expert labelling (Snow et al., 2008) It has been utilised in the generation of image an- notations to train computer vision systems (Raykar et al., 2010), to provide the linguistic annotations used for Natural Language Processing (NLP) tasks (Snow et al., 2008), and has also been used to collect the relevant judgments needed to optimize search engines (Alonso, 2013) . It is a well known fact that crowd-sourced labels are known to be associated with noise and errors as a result of the annotations being provided by anno- tators with uneven expertise and dedication which can result in the compromise of practical applica- tions that uses such data (Zhang et al., 2016). This paper therefore seeks to apply a novel approach to minimize and mitigate the noise and errors in crowd sourced labels. The aim is to …

Content truncated to 500 words