Aggregating Crowdsourced and Automatic Judgments to Scale Up a Corpus of Anaphoric Reference for Fiction and Wikipedia Texts

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 767–781 May 2-6, 2023 ©2023 Association for Computational Linguistics Aggregating Crowdsourced and Automatic Judgments to Scale Up a Corpus of Anaphoric Reference for Fiction and Wikipedia Texts Juntao Yu1, Silviu Paun2∗, Maris Camilleri3, Paloma Carretero Garcia3, Jon Chamberlain1, Udo Kruschwitz 4 and Massimo Poesio3 1University of Essex, UK; 2Amazon Research, Romania; 3Queen Mary University of London, UK; 4University of Regensburg, Germany. j.yu@essex.ac.uk; silviupn@amazon.com; mcamil@essex.ac.uk;pcarre@essex.ac.uk; jchamb@essex.ac.uk; udo.kruschwitz@ur.de; m.poesio@qmul.ac.uk; Abstract Although several datasets annotated for anaphoric reference / coreference exist, even the largest such datasets have limitations in terms of size, range of domains, coverage of anaphoric phenomena, and size of documents included. Yet, the approaches proposed to scale up anaphoric annotation haven’t so far resulted in datasets overcoming these limitations. In this paper, we introduce a new release of a corpus for anaphoric reference labelled via a game- with-a-purpose. This new release1 is compa- rable in size to the largest existing corpora for anaphoric reference due in part to substantial activity by the players, in part thanks to the use of a new resolve-and-aggregate paradigm to ‘complete’ markable annotations through the combination of an anaphoric resolver and an ag- gregation method for anaphoric reference. The proposed method could be adopted to greatly speed up annotation time in other projects in- volving games-with-a-purpose. In addition, the corpus covers genres for which no compara- ble size datasets exist (Fiction and Wikipedia); it covers singletons and non-referring expres- sions; and it includes a substantial number of long documents (> 2K in length). 1 Introduction Many resources annotated for anaphoric reference / coreference exist; but even the largest such datasets, such as ONTONOTES (Pradhan et al., 2012), have limitations. The largest resources are still medium scale (e.g., ONTONOTES (Pradhan et al., 2012) is 1.5M tokens, as is CRAFT (Cohen et al., 2017)). They only cover a limited range of domains, pri- marily news (as in ONTONOTES) and scientific ar- ticles (as in CRAFT), and models trained on these datasets have been shown not to generalize well to other domains (Xia and Durme, 2021).2 The range ∗Work was done prior to joining Amazon research. 1The corpus is available at https://github.com/ dali-ambiguity/Phrase-Detectives-Corpus-3.0 2The largest existing corpus for English, the 10M words PRECO (Chen et al., 2018), consists of language learning texts, of anaphoric phenomena covered is also narrow (Poesio et al., 2016). Several proposals have been made to scale up anaphoric annotation in terms of size, range of do- mains, and phenomena covered proposed, includ- ing automatic data augmentation (Emami et al., 2019; Gessler et al., 2020; Aloraini and Poesio, 2021), and crowdsourcing combined with active learning (Laws et al., 2012; Li et al., 2020; Yuan et al., 2022) or through Games-With-A-Purpose (Chamberlain et al., 2008; Hladká et al., 2009; Bos et al., 2017; Kicikoglu et al., 2019). However, the largest existing anaphoric corpora created using Games-With-A-Purpose (e.g., (Poesio et al., 2019)) are still smaller than the largest resources created with …

Content truncated to 500 words