A Bayesian Approach for Sequence Tagging with Crowds

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1093–1104, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1093 A Bayesian Approach for Sequence Tagging with Crowds Edwin Simpson and Iryna Gurevych Ubiquitous Knowledge Processing Lab Department of Computer Science Technische Universit¨at Darmstadt https://www.ukp.tu-darmstadt.de {simpson,gurevych}@ukp.informatik.tu-darmstadt.de Abstract Current methods for sequence tagging, a core task in NLP, are data hungry, which motivates the use of crowdsourcing as a cheap way to obtain labelled data. However, annotators are often unreliable and current aggregation meth- ods cannot capture common types of span an- notation errors. To address this, we propose a Bayesian method for aggregating sequence tags that reduces errors by modelling sequen- tial dependencies between the annotations as well as the ground-truth labels. By taking a Bayesian approach, we account for uncer- tainty in the model due to both annotator er- rors and the lack of data for modelling anno- tators who complete few tasks. We evaluate our model on crowdsourced data for named entity recognition, information extraction and argument mining, showing that our sequential model outperforms the previous state of the art. We also find that our approach can re- duce crowdsourcing costs through more effec- tive active learning, as it better captures un- certainty in the sequence labels when there are few annotations. 1 Introduction Current methods for sequence tagging, a core task in NLP, use deep neural networks that require tens of thousands of labelled documents for training (Ma and Hovy, 2016; Lample et al., 2016). This presents a challenge when facing new domains or tasks, where obtaining labels is often time- consuming or costly. Labelled data can be ob- tained cheaply by crowdsourcing, in which large numbers of untrained workers annotate documents instead of more expensive experts. For sequence tagging, this results in multiple sequences of unre- liable labels for each document. Probabilistic methods for aggregating crowd- sourced data have been shown to be more accu- rate than simple heuristics such as majority vot- ing (Raykar et al., 2010; Sheshadri and Lease, 2013; Rodrigues et al., 2013; Hovy et al., 2013). However, existing methods for aggregating se- quence labels cannot model dependencies between the annotators’ labels (Rodrigues et al., 2014; Nguyen et al., 2017) and hence do not account for their effect on annotator noise and bias. In this paper, we remedy this by proposing a sequential annotator model and applying it to tasks that fol- low a beginning, inside, outside (BIO) scheme, in which the first token in a span of type ‘x’ is la- belled ‘B-x’, subsequent tokens are labelled ‘I-x’, and tokens outside spans are labelled ‘O’. When learning from noisy or small datasets, commonly-used methods based on maximum like- lihood estimation may produce over-confident pre- dictions (Xiong et al., 2011; Srivastava et al., 2014). In contrast, Bayesian inference accounts for model uncertainty when making predictions. Unlike alternative methods that optimize the val- ues for model parameters, Bayesian …

Content truncated to 500 words