Phrase Detectives Corpus 1.0 Crowdsourced Anaphoric Coreference.

Phrase Detectives Corpus 1.0 Crowdsourced Anaphoric Coreference Jon Chamberlain, Massimo Poesio, Udo Kruschwitz School of Computer Science and Electronic Engineering University of Essex {jchamb,poesio,udo}@essex.ac.uk Abstract Natural Language Engineering tasks require large and complex annotated datasets to build more advanced models of language. Corpora are typically annotated by several experts to create a gold standard; however, there are now compelling reasons to use a non-expert crowd to annotate text, driven by cost, speed and scalability. Phrase Detectives Corpus 1.0 is an anaphorically-annotated corpus of encyclopedic and narrative text that contains a gold standard created by multiple experts, as well as a set of annotations created by a large non-expert crowd. Analysis shows very good inter-expert agreement (κ = .88 − .93) but a more variable baseline crowd agreement (κ = .52 − .96). Encyclopedic texts show less agreement (and by implication are harder to annotate) than narrative texts. The release of this corpus is intended to encourage research into the use of crowds for text annotation and the development of more advanced, probabilistic language models, in particular for anaphoric coreference. Keywords: anaphoric coreference, anaphora, annotation, crowdsourcing, gwap, games-with-a-purpose, phrase detectives, corpora 1. Introduction A revolution in the way tasks can be completed occurred when it was proposed to take a job traditionally performed by a designated employee and outsource it to an undefined large group of Internet users. This approach, called crowd- sourcing (Howe, 2008), changed traditional thinking be- hind methods for language resource creation and made new tasks possible that were previously inconceivable due to cost or labour limitations. For example, microworking is now a common crowdsourcing approach to create small to medium-sized language resources by engaging a crowd us- ing small payments (Snow et al., 2008). An alternative ap- proach is to use a game-with-a-purpose (GWAP) to ag- gregate data from non-expert players, who are motivated by entertainment, to create collective decisions similar to those from an expert (von Ahn, 2006). Phrase Detectives, an interactive online game for creating anaphorically-annotated corpora, is an illustration of the GWAP approach for creating large-scale resources. The Phrase Detectives corpus differs from existing corpora for anaphora in two key respects: (i) it covers genres for which no other data are available, including encyclopedic and nar- rative text; and (ii) multiple solutions (or interpretations) are collected per task. This paper briefly describes the game and annotation scheme, before describing in more detail the measures of quality used and an analysis of a subset of the corpus which has been made available to the language re- source community. 2. Related Work Generally speaking, a game-based crowdsourcing approach uses entertainment rather than financial payment to moti- vate participation. GWAPs come in many forms; they tend to be graphically rich, with simple interfaces, and give the player an experience of progression by scoring points, be- ing assigned levels and recognising their effort. Systems are required to control the behaviour of players: to encour- age them to concentrate on the tasks and to discourage them from malicious behaviour. …

Content truncated to 500 words