Comparing Bayesian Models of Annotation

Comparing Bayesian Models of Annotation Silviu Paun1 Bob Carpenter2 Jon Chamberlain3 Dirk Hovy4 Udo Kruschwitz3 Massimo Poesio1 1School of Electronic Engineering and Computer Science, Queen Mary University of London 2Department of Statistics, Columbia University 3School of Computer Science and Electronic Engineering, University of Essex 4Department of Marketing, Bocconi University Abstract The analysis of crowdsourced annotations in natural language processing is con- cerned with identifying (1) gold standard labels, (2) annotator accuracies and biases, and (3) item difficulties and error patterns. Traditionally, majority voting was used for 1, and coefficients of agreement for 2 and 3. Lately, model-based analysis of corpus annotations have proven better at all three tasks. But there has been relatively little work comparing them on the same datasets. This paper aims to fill this gap by ana- lyzing six models of annotation, covering different approaches to annotator ability, item difficulty, and parameter pooling (tying) across annotators and items. We evaluate these models along four aspects: comparison to gold labels, predictive accu- racy for new annotations, annotator char- acterization, and item difficulty, using four datasets with varying degrees of noise in the form of random (spammy) annotators. We conclude with guidelines for model selec- tion, application, and implementation. 1 Introduction The standard methodology for analyzing crowd- sourced data in NLP is based on majority vot- ing (selecting the label chosen by the majority of coders) and inter-annotator coefficients of agree- ment, such as Cohen’s κ (Artstein and Poesio, 2008). However, aggregation by majority vote implicitly assumes equal expertise among the annotators. This assumption, though, has been re- peatedly shown to be false in annotation prac- tice (Poesio and Artstein, 2005; Passonneau and Carpenter, 2014; Plank et al., 2014b). Chance- adjusted coefficients of agreement also have many shortcomings—for example, agreements in mis- take, overly large chance-agreement in datasets with skewed classes, or no annotator bias correc- tion (Feinstein and Cicchetti, 1990; Passonneau and Carpenter, 2014). Research suggests that models of annotation can solve these problems of standard practices when applied to crowdsourcing (Dawid and Skene, 1979; Smyth et al., 1995; Raykar et al., 2010; Hovy et al., 2013; Passonneau and Carpenter, 2014). Such probabilistic approaches allow us to characterize the accuracy of the annotators and correct for their bias, as well as account- ing for item-level effects. They have been shown to perform better than non-probabilistic alterna- tives based on heuristic analysis or adjudication (Quoc Viet Hung et al., 2013). But even though a large number of such models has been proposed (Carpenter, 2008; Whitehill et al., 2009; Raykar et al., 2010; Hovy et al., 2013; Simpson et al., 2013; Passonneau and Carpenter, 2014; Felt et al., 2015a; Kamar et al., 2015; Moreno et al., 2015, in- ter alia), it is not immediately obvious to potential users how these models differ or, in fact, how they should be applied at all. To our knowledge, the literature comparing models of annotation is lim- ited, focused exclusively on synthetic data (Quoc Viet Hung et al., 2013) or using publicly available implementations …

Content truncated to 500 words