A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation

Proceedings of NAACL-HLT 2019, pages 1778–1789 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1778 A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation Massimo Poesio Queen Mary University m.poesio@qmul.ac.uk Jon Chamberlain University of Essex jchamb@essex.ac.uk Udo Kruschwitz University of Essex udo@essex.ac.uk Silviu Paun Queen Mary University s.paun@qmul.ac.uk Alexandra Uma Queen Mary University a.n.uma@qmul.ac.uk Juntao Yu Queen Mary University juntao.cn@gmail.com Abstract We present a corpus of anaphoric information (coreference) crowdsourced through a game- with-a-purpose. The corpus, containing anno- tations for about 108,000 markables, is one of the largest corpora for coreference for English, and one of the largest crowdsourced NLP cor- pora, but its main feature is the large num- ber of judgments per markable: 20 on aver- age, and over 2.2M in total. This character- istic makes the corpus a unique resource for the study of disagreements on anaphoric in- terpretation. A second distinctive feature is its rich annotation scheme, covering single- tons, expletives, and split-antecedent plurals. Finally, the corpus also comes with labels in- ferred using a recently proposed probabilistic model of annotation for coreference. The la- bels are of high quality and make it possible to successfully train a state of the art corefer- ence resolver, including training on singletons and non-referring expressions. The annotation model can also result in more than one label, or no label, being proposed for a markable, thus serving as a baseline method for automatically identifying ambiguous markables. A prelimi- nary analysis of the results is presented. 1 Introduction A number of datasets for anaphora resolution / coreference now exist (Poesio et al., 2016), includ- ing ONTONOTES that has been the de facto stan- dard since the CONLL shared tasks in 2011 and 2012 (Pradhan et al., 2012), and the just intro- duced and very substantial PRECO corpus (Chen et al., 2018). None of these datasets however take into account the research challenging the idea that a ‘gold standard’ interpretation can be obtained through adjudication, in particular for anaphora (Poesio and Artstein, 2005b; Wong and Lee, 2013; Aroyo and Welty, 2015). Virtually every project devoted to large-scale annotation of discourse or semantic phenomena has reached the conclusion that genuine disagreements are widespread. This has long been known for anaphora (Poesio and Artstein, 2005b; Versley, 2008; Recasens et al., 2011) (see also the analysis of disagreements in ONTONOTES in (Pradhan et al., 2012)) and word- senses (Passonneau et al., 2012), but more recent work has provided evidence that disagreements are frequent for virtually every aspect of language interpretation, not just in subjective tasks such as sentiment analysis (Kenyon-Dean et al., 2018), but even in the case of tasks such as part-of-speech tagging (Plank et al., 2014). In fact, researchers in the CrowdTruth project view disagreement as positive, arguing that “disagreement is signal, not noise” (Aroyo and Welty, 2015). In this paper we present what to our knowledge is the largest cor- pus containing alternative anaphoric judgments: 20.6 judgments per markable on average (up to 90 judgments …

Content truncated to 500 words