Modelling Instance-Level Annotator Reliability for Natural Language Labelling Tasks

Proceedings of NAACL-HLT 2019, pages 2873–2883 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2873 Modelling Instance-Level Annotator Reliability for Natural Language Labelling Tasks Maolin Li1,2, Arvid Fahlstr¨om Myrman2, Tingting Mu2 and Sophia Ananiadou1,2 1National Centre for Text Mining, University of Manchester, United Kingdom 2School of Computer Science, University of Manchester, United Kingdom {maolin.li, arvidfm, tingting.mu, sophia.ananiadou}@manchester.ac.uk Abstract When constructing models that learn from noisy labels produced by multiple annotators, it is important to accurately estimate the reli- ability of annotators. Annotators may provide labels of inconsistent quality due to their vary- ing expertise and reliability in a domain. Pre- vious studies have mostly focused on estimat- ing each annotator’s overall reliability on the entire annotation task. However, in practice, the reliability of an annotator may depend on each specific instance. Only a limited num- ber of studies have investigated modelling per- instance reliability and these only considered binary labels. In this paper, we propose an un- supervised model which can handle both bi- nary and multi-class labels. It can automat- ically estimate the per-instance reliability of each annotator and the correct label for each instance. We specify our model as a proba- bilistic model which incorporates neural net- works to model the dependency between la- tent variables and instances. For evaluation, the proposed method is applied to both syn- thetic and real data, including two labelling tasks: text classification and textual entail- ment. Experimental results demonstrate our novel method can not only accurately estimate the reliability of annotators across different in- stances, but also achieve superior performance in predicting the correct labels and detecting the least reliable annotators compared to state- of-the-art baselines.1 1 Introduction In many natural language processing (NLP) ap- plications, the performance of supervised machine learning models depends on the quality of the cor- pus used to train the model. Traditionally, la- bels are collected from multiple annotators/experts 1Code is available at https://github.com/ createmomo/instance-level-reliability who are assumed to provide reliable labels. How- ever, in reality, these experts may have varying levels of expertise depending on the domains, and thus may disagree on labelling in certain cases (Aroyo and Welty, 2013). A rapid and cost- effective alternative is to obtain labels through crowdsourcing (Snow et al., 2008; Poesio et al., 2013, 2017). In crowdsourcing, each instance is presented to multiple expert or non-expert anno- tators for labelling. However, labels collected in this manner could be noisy, since some annotators could produce a significant number of incorrect la- bels. This may be due to differing levels of exper- tise, lack of financial incentive and interest (Poesio et al., 2017), as well as the tedious and repetitive nature of the annotation task (Raykar et al., 2010; Bonald and Combes, 2017). Thus, in order to ensure the accuracy of the la- belling and the quality of the corpus, it is crucial to estimate the reliability of the annotators automati- cally without human intervention. Previous studies have mostly focused on evalu- ating the annotators’ overall reliability …

Content truncated to 500 words