Empirical Analysis of Aggregation Methods for Collective Annotation

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1533–1542, Dublin, Ireland, August 23-29 2014. Empirical Analysis of Aggregation Methods for Collective Annotation Ciyang Qing, Ulle Endriss, Raquel Fern´andez and Justin Kruger Institute for Logic, Language and Computation University of Amsterdam {qciyang | justin.g.kruger}@gmail.com {ulle.endriss | raquel.fernandez}@uva.nl Abstract We investigate methods for aggregating the judgements of multiple individuals in a linguistic annotation task into a collective judgement. We define several aggregators that take the relia- bility of annotators into account and thus go beyond the commonly used majority vote, and we empirically analyse their performance on new datasets of crowdsourced data. 1 Introduction Human annotation of linguistic resources has become indispensable in computational linguistics, es- pecially with regards to semantic and pragmatic information, which is yet beyond the reach of robust automatic labelling. Most annotation campaigns involve a small group of trained annotators who may not always agree on their judgements. The reliability of the annotation is typically assessed by quan- tifying the level of inter-annotator agreement, while the final annotation to be released is consensuated amongst experts. In recent years, however, crowdsourcing methods such Amazon’s Mechanical Turk (AMT) have shaken up this scenario by making it possible to rapidly recruit large numbers of untrainned annotators at a low cost. This offers great opportunities—in particular, if we consider that the community of speakers is the highest authority regarding linguistic knowledge—but also creates several challenges: amongst others, how to obtain good quality annotations from untrainned and unmonitored individuals, and how to combine large numbers of possibly conflicting judgements into a single joint annotation. In this paper we focus on the latter challenge. Our aim is to investigate and empirically test methods for aggregating the judgements of large numbers of individuals in a linguistic annotation task conducted via crowdsourcing into a collective judgement. Most researchers who turn to crowdsourcing to collect data use majority voting to combine the par- ticipants’ responses (Sayeed et al., 2011; Zarcone and R¨ud, 2012; Venhuizen et al., 2013). Although in the limit it makes sense to take the judgement of the majority as reflecting the view of the community, in practice we cannot reach out to the full population of speakers, which means that the possible biases amongst the participants we manage to recruit may distort the outcome. Also, given the nature of crow- sourcing (rewarding speed rather than quality), some participants may not respond truthfully according to their intuitions as speakers. To address these issues, we propose aggregation methods that go beyond majority voting by taking into account the reliability of individual annotators at the time of aggregation.1 Our approach is related to existing work on analysing the quality of annotated data by examining, for instance, (dis)agreement patterns amongst annotators (Bhardwaj et al., 2010; Peldszus and Stede, 2013; Ramanath et al., 2013). However, while the main aim of this kind of studies is to gain insight into the difficulty of an annotation task or into the feasibility of using untrainned annotators …

Content truncated to 500 words