Sequence Effects in Crowdsourced Annotations

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2860–2865 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Sequence Effects in Crowdsourced Annotations Nitika Mathur Timothy Baldwin Trevor Cohn School of Computing and Information Systems The University of Melbourne Victoria 3010, Australia nmathur@student.unimelb.edu.au {tbaldwin,tcohn}@unimelb.edu.au Abstract Manual data annotation is a vital compo- nent of NLP research. When designing annotation tasks, properties of the anno- tation interface can lead to unintentional artefacts in the resulting dataset, biasing the evaluation. In this paper, we explore sequence effects where annotations of an item are affected by the preceding items. Having assigned one label to an instance, the annotator may be less (or more) likely to assign the same label to the next. Dur- ing rating tasks, seeing a low quality item may affect the score given to the next item either positively or negatively. We see clear evidence of both types of effects us- ing auto-correlation studies over three dif- ferent crowdsourced datasets. We then recommend a simple way to minimise se- quence effects. 1 Introduction NLP research relies heavily on annotated datasets for training and evaluation. The design of the an- notation task can influence the decisions made by annotators in subtle ways: besides the actual fea- tures of the instance being annotated, annotators are also influenced by factors such as the user in- terface, wording of the question, and familiarity with the task or domain. When collecting NLP annotations, care is usu- ally taken to ensure that the annotations are of high quality, through careful design of label sets, anno- tation guidelines and training of annotators (Hovy et al., 2006), methods for aggregating annotations (Passonneau and Carpenter, 2014), and intuitive user interfaces (Stenetorp et al., 2012). Crowdsourcing has emerged as a cheaper, faster alternative to expert NLP annotations (Snow et al., 2008; Callison-Burch and Dredze, 2010; Graham et al., 2017), although it entails additional effort to filter out unskilled or opportunistic workers, e.g. through the collection of redundant repeated judgements for each instance, or including some trap questions with known answers (Callison- Burch and Dredze, 2010; Hoßfeld et al., 2014). In most annotation exercises, the order of pre- sentation of instances is randomised to remove bias due to similarities in topic, style and vocabu- lary (Koehn and Monz, 2006; Bojar et al., 2016). When crowdsourcing judgements, the normal practise (as used in the datasets we analyse) is for the item ordering to be randomised in creating a “HIT” (i.e. a single collection of items presented to a crowdworker for judgement), and then to have each HIT annotated by multiple workers, for qual- ity control purposes. The order of items is gen- erally fixed across all annotators of an individual HIT (Snow et al., 2008; Graham et al., 2017). In this paper, we show that worker scores are affected by sequence bias, whereby the order of presentation can affect individuals’ assessment of an item. Since all workers see the instances in the same order, this affects …

Content truncated to 500 words