Collective Annotation of Linguistic Resources: Basic Principles and a Formal Model

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 539–549, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Collective Annotation of Linguistic Resources: Basic Principles and a Formal Model Ulle Endriss and Raquel Fern´andez Institute for Logic, Language & Computation University of Amsterdam {ulle.endriss|raquel.fernandez}@uva.nl Abstract Crowdsourcing, which offers new ways of cheaply and quickly gathering large amounts of information contributed by volunteers online, has revolutionised the collection of labelled data. Yet, to create annotated linguistic resources from this data, we face the challenge of having to combine the judgements of a potentially large group of annotators. In this paper we investigate how to aggregate individual annotations into a single collective anno- tation, taking inspiration from the field of social choice theory. We formulate a gen- eral formal model for collective annotation and propose several aggregation methods that go beyond the commonly used major- ity rule. We test some of our methods on data from a crowdsourcing experiment on textual entailment annotation. 1 Introduction In recent years, the possibility to undertake large- scale annotation projects with hundreds or thou- sands of annotators has become a reality thanks to online crowdsourcing methods such as Amazon’s Mechanical Turk and Games with a Purpose. Al- though these techniques open the door to a true revolution for the creation of annotated corpora, within the computational linguistics community there so far is no clear understanding of how the so-called “wisdom of the crowds” could or should be used to develop useful annotated linguistic re- sources. Those who have looked into this increas- ingly important issue have mostly concentrated on validating the quality of multiple non-expert an- notations in terms of how they compare to ex- pert gold standards; but they have only used sim- ple aggregation methods based on majority voting to combine the judgments of individual annotators (Snow et al., 2008; Venhuizen et al., 2013). In this paper, we take a different perspective and instead focus on investigating different aggrega- tion methods for deriving a single collective an- notation from a diverse set of judgments. For this we draw inspiration from the field of social choice theory, a theoretical framework for combining the preferences or choices of several individuals into a collective decision (Arrow et al., 2002). Our aim is to explore the parallels between the task of ag- gregating the preferences of the citizens participat- ing in an election and the task of combining the expertise of speakers taking part in an annotation project. Our contribution consists in the formula- tion of a general formal model for collective an- notation and, in particular, the introduction of sev- eral families of aggregation methods that go be- yond the commonly used majority rule. The remainder of this paper is organised as fol- lows. In Section 2 we introduce some basic termi- nology and argue that there are four natural forms of collective annotation. We then focus on one of them and present a formal model for it in Sec- tion …

Content truncated to 500 words